Deciding bronze layer retention when a show runs once a year

The question arrives from the wrong direction, which is why it usually goes unanswered for years. Somebody in legal asks how long the company keeps registration data. The data team answers that everything is in bronze and bronze is immutable, which they mean as reassurance. Legal hears that nothing is ever deleted.

Bronze layer retention is the point where a filing convention meets an obligation, and for a show that runs once a year the arithmetic is unusual enough that borrowed defaults from a daily pipeline are the wrong answer in both directions.

What does replay actually need?

The reason to keep raw files at all is the ability to rebuild everything above them. That is the whole justification for a layer that never changes, and it sets a hard floor under any retention policy.

The floor is the comparison window. A portfolio report that shows five editions of a show needs to be rebuildable across five editions, because the day somebody corrects the definition of verified attendance, all five numbers move together or the chart is nonsense. Five editions of an annual show means the oldest useful raw file is five years old, and four year-over-year comparisons sit inside that window.

That is a much longer floor than most pipelines carry. A business loading daily can rebuild its comparison window from ninety days of raw data. An exhibition needs 1,825 days to hold the same number of comparison points, and there is no way to shorten it, because the events being compared happened a year apart.

What replay needs is the bytes. Not a parsed table, not the cleaned version, not a row count. The file as the vendor sent it, because the parsing is the part you are most likely to want to redo.

The second clock, which is not yours to set

Article 5(1)(e) of the GDPR, Regulation 2016/679, states the storage limitation principle, requiring that personal data be "kept in a form which permits identification of data subjects for no longer than is necessary for the purposes for which" it is processed, with a carve out for archiving, research and statistical purposes under Article 89(1).

The operative words are necessary and purposes. Neither is a number, and neither is a data engineer's to decide. What the data team owns is the technical shape of the answer: knowing which files contain personal data, being able to say exactly what is in each one, and being able to expire a given edition cheaply when somebody sets a date.

Of the six raw files a typical edition lands, the registration export, the badge scan export and the lead retrieval export are dense with personal data. The session programme carries speaker details. The exhibitor contract list is mostly company data with contact names attached. The floor plan allocation usually has none at all. Applying a single retention period across all six because they arrived in the same folder is the common shortcut, and it either keeps personal data longer than anyone intended or throws away the floor plan for no reason.

Retention is decided per file, and the policy is the shorter of the two clocks for that file.

Partition by edition so expiry costs nothing

If a retention date is ever going to be enforced, deleting has to be trivial, because a delete that requires an engineer to write a careful WHERE clause under time pressure is a delete that does not happen.

The PostgreSQL 18 documentation, current in 2026, is explicit about the mechanism. "Bulk loads and deletes can be accomplished by adding or removing partitions, if the usage pattern is accounted for in the partitioning design. Dropping an individual partition using DROP TABLE, or doing ALTER TABLE DETACH PARTITION, is far faster than a bulk operation. These commands also entirely avoid the VACUUM overhead caused by a bulk DELETE." On removing history specifically, the same documentation says dropping the partition "can very quickly delete millions of records because it doesn't have to individually delete every record".

So partition bronze by edition and by source file. A portfolio of 8 shows with 5 years of history and 6 files per edition is 5 times 8 times 6, or 240 partitions. Expiring the 2021 edition of one show is six detach statements.

That partition count is comfortable. The PostgreSQL 18 documentation notes that "the query planner is generally able to handle partition hierarchies with up to a few thousand partitions fairly well, provided that typical queries allow the query planner to prune all but a small number of partitions". Twenty years of the same portfolio would be 960 partitions and still inside that guidance, and no exhibitions group is going to be keeping raw badge scan files for twenty years anyway.

The detach form is worth preferring over the drop form during the first year of any policy. Detaching leaves the data as a standalone table that is out of the warehouse and still recoverable, which converts an irreversible decision into a reversible one for the price of some disk.

Is storage cost a real reason to expire bronze?

Almost never, and the argument gets made constantly, so it is worth doing the sums.

One edition's raw files are small. 38,900 badge scan rows as a delimited export at roughly 120 bytes a row is about 4.7 megabytes. 12,400 registration rows with the full form response at roughly 600 bytes a row is about 7.4 megabytes. Add the other four files and one edition of one show lands somewhere near 15 megabytes of raw bronze. Forty editions across the portfolio is about 600 megabytes, uncompressed, and these files compress well.

Six hundred megabytes is less than the photography from a single show. Anyone proposing to delete raw registration exports to save storage is optimising a cost that rounds to zero, and is doing so at the price of the rebuild property that justified the layer in the first place.

The real reasons to expire bronze are the legal clock, and the reduction in what a breach would expose. Both are good reasons. Storage is not one, and letting the storage argument carry the decision produces a policy nobody can defend when the question is asked properly.

Expiring columns instead of editions

The two clocks conflict most often at the six or seven year mark, where the analytical case for keeping an edition has weakened and the legal case for deleting it has not yet become obvious. There is a middle option that most policies omit.

Retention can be applied per column. The identifying columns in a registration export are a small subset of it: name, email, phone, employer contact details, sometimes a postal address. The rest of the row is job function, seniority band, country, registration type, channel, timestamps and form responses. Expiring the first group and keeping the second leaves a file that still supports every trend, mix and pacing rebuild while no longer permitting identification.

For an older edition that is a good trade. A 2019 registration file is worth keeping as shape and worth very little as a contact list, and a contact list is precisely the part that carries risk. The mechanics are a rewrite of the partition with the identifying columns replaced by a salted hash or dropped outright, which does break the original checksum, so record that the transformation happened and when.

Whether this counts as anonymisation in the legal sense is a judgement for whoever owns the privacy notice, and it depends on how re-identifiable the remainder is. The data team's job is to make the option available and to be precise about what was removed.

Why a kept file is not automatically a usable one

Immutability preserves the bytes and preserves nothing about your ability to read them.

Three things rot around a file that itself never changes. The export format, because the vendor redesigned their reporting module between editions and the 2022 file has columns the 2025 file does not. The encoding, because somebody's export defaulted to a Windows code page one year and UTF-8 the next. And the loader, because the code that parsed the 2022 file was deleted in a refactor two years ago.

The fix is to store the loader alongside the file, or at minimum to store the schema you observed at load time: column names, order, encoding, row count and a checksum. That metadata is a few hundred bytes per file and it is the difference between an archive and an input. Where the formats have already diverged across editions, reconciling them is a backfill problem with its own cost profile.

A related discipline is proving the rebuild works rather than assuming it. An untested rebuild path is a hypothesis, and the mechanics of keeping one exercised belong to the replayable pipeline design.

Where the policy breaks down

Two situations defeat a clean per edition retention rule, and both are common.

The first is the derived record that outlives its source. A person's golden record in silver was built from registration rows across five editions. Expire the 2021 bronze partition and the golden record still carries attributes that came from it, so the personal data has not actually gone. Retention that stops at bronze is a partial deletion dressed as a complete one, and the honest version has to reach into silver, which is far harder because silver rows are merged from many sources.

The second is the request that arrives out of band. An individual asks for their data to be erased, and bronze is immutable by design. Detaching a whole edition partition to remove one person is not proportionate, and rewriting the file breaks the checksum that made it evidence. Most teams land on a suppression list plus a documented process, and the important thing is deciding that in advance instead of during the request.

Neither is a reason to abandon immutability. Both are reasons to write down what bronze does and does not guarantee before somebody assumes it guarantees more.

Open the folder holding your most recent edition, list the files, and write one line per file naming what personal data it contains and which clock governs it. That inventory takes an afternoon, it is the thing legal has actually been asking for, and it belongs in the data platform conventions rather than in one person's head.