Building a replayable data pipeline you can rebuild from raw files

Someone asks why last year's verified attendance in the board pack says 11,204 and the number on the dashboard today says 11,180. Nobody changed the fact table. Nobody restated anything. The pipeline has been touched eleven times since, by four people, and no one of those changes was supposed to move a closed edition.

Replayable data pipeline design is the property that turns that conversation from an archaeology project into a diff. If you can drop every derived table and rebuild the warehouse from the raw files, then the numbers you get back are what the current code believes about history, and anything that differs from what you published is a defect with a location.

Most event data teams cannot do this, and the reason is rarely the compute bill. It is that somewhere in the pipeline, a table was edited in place.

What replayable actually means

Maxime Beauchemin set out the paradigm in 2018 in his essay on functional data engineering, and the framing has held up better than most things written about data pipelines that decade. His statement of the goal is the one to hold onto: "Given a persistent immutable staging area and pure tasks, in theory it's possible to recompute the state of the entire warehouse from scratch...and get to the exact same state."

Two conditions sit inside that sentence, and both have to be true.

The first is the staging area. Beauchemin describes it as "the conceptual loading dock of your data warehouse" and says "you'll want to accumulate and persist all of your source data there, and keep it unchanged forever." For a show that runs annually, that means the badge vendor's export from 2022 is still sitting exactly as it arrived, with the column that no longer exists and the status code nobody documented.

The second is the transformation. "A pure task should be deterministic and idempotent, meaning that it will produce the same result every time it runs or re-runs", and "A pure task should always fully overwrite a partition as its output." A task that appends, or that patches rows in place, has destroyed the ability to reproduce itself the moment it runs. Getting one task to that standard is the subject of idempotent warehouse loads, and replay is what you get once every task in the graph clears the bar at the same time.

Why does an annual show make replay harder?

An event pipeline gets one real execution a year per show, which changes the economics of everything.

A daily pipeline that drifts gets caught within a week, because tomorrow's run disagrees with today's and somebody is looking. An annual pipeline that drifts gets caught in eleven months, by which point the code has moved on, the person who wrote it has moved on, and the only witness is a number in a PDF.

Replay is the substitute for that daily feedback. Rebuilding 40 editions from raw files gives you 40 independent checks of code that would otherwise be exercised once. It is the closest an annual cadence gets to the thing continuous pipelines get for free, and it costs a rebuild rather than a year.

The 40 partition rebuild, worked through

Here is the exercise in the shape it usually takes. Eight shows, five years, so 40 editions in bronze. Point the pipeline at an empty target schema and rebuild every edition from the raw files, newest first so that the editions people actually ask about land early.

At roughly two minutes an edition, the whole portfolio rebuilds in about 80 minutes. That number matters more than it looks, because a rebuild that takes 80 minutes is something you can run on a Tuesday afternoon on a hunch, and a rebuild that takes two days is something that never gets run at all.

Then compare. Registration counts came back identical for all 40 editions, 12,400 for the edition in question against 12,400 published. Verified attendance came back 11,180 against 11,204 published, a difference of 24 rows on a base of 11,204, which is 0.21 per cent. Small enough that no dashboard would have flagged it and no stakeholder would have queried it.

The 24 rows turned out to be badge scans within 90 minutes of doors closing on the final day, and the transformation classifying them as in-session used the current date to decide whether an edition was closed. Run in 2025, the expression returned one answer. Run in 2026, it returned another. Nothing about the data changed. The code was reading the clock, which makes it impure in exactly Beauchemin's sense, and the rebuild exposed it because a rebuild is the only test that runs the old data through the new day.

Where determinism leaks

The clock is the most common leak and it has company. In an event warehouse the recurring offenders are worth naming, because each one has a specific fix.

• The current date or current timestamp. Pass the edition's own reference dates in as parameters and let the function be pure with respect to them.
• A dimension that has been overwritten since. If a type 1 attribute was updated in place, the rebuild joins today's company name to a 2022 fact and gets a different answer than the original run did. The company dimension needs its history preserved before replay means anything, which is the slowly changing dimension problem arriving in a new costume.
• Row order. Any window function with a tie in its ORDER BY can produce different survivors on different runs. Add a deterministic tiebreak column and the ambiguity disappears.
• Random or hash-based sampling that is not seeded.
• A lookup file someone edited. Mapping tables that live in a spreadsheet and get corrected quietly are raw inputs pretending to be configuration. They belong in bronze with a version, so a rebuild can pick the version that was current at load time.

What does replay buy you that a test does not?

Tests check the cases you thought of. Replay checks the cases the last five years actually contained.

A unit test with a hand built fixture asks whether the logic still handles a duplicate scan and a null company. A rebuild asks whether the logic, applied to 1.5 million real rows collected under nine different registration platforms, still produces the numbers you sent to a board. Those are different questions and the second one is unavailable to any test you can write in advance.

Replay also gives you something operationally useful that tests do not: permission to change things. Tobiko Data's SQLMesh documentation, current in 2026, describes restatement as re-evaluating a model for a given time range "even though the model definition has not changed", and frames plans around the idea that "all changes can be reviewed and verified before they are applied to the data warehouse." A team that can rebuild any edition on demand can accept a correction to a five year old source file without a meeting about whether it is worth it.

Comparing every published measure across every edition after a refactor is a related discipline with its own tooling, and it belongs to regression testing a refactor rather than here. Replay proves you can reproduce the past. The diff tells you whether a specific change was safe.

What replay costs you in design freedom

The bill for replayability is paid in things you are no longer allowed to do.

You cannot update a fact row in place. You cannot let a transformation read a table that a later step will modify. You cannot store a value that was computed from a source you did not keep. You cannot accept a source feed that only exposes current state with no history, unless you snapshot it yourself on arrival, which is more work than it sounds because the snapshot needs its own retention and its own raw layer discipline.

You also pay in storage, and this is where the argument usually gets lost, because storage is the one line item people can see. Raw exports for a portfolio of this size are small against any modern warehouse bill. The comparison worth making is against one person spending three days reconciling a board number by hand, which happens at least once a year in every organisation I have worked with.

The constraint most teams resent is the last one, and it is the one I would hold hardest. A source that only exposes current state has to be snapshotted on arrival or it silently breaks replay for everything computed from it, and the moment a data platform accepts one such source without snapshotting it, the guarantee stops being a guarantee and becomes a claim about the parts of the warehouse somebody happens to remember.

The limit

Replay reproduces the pipeline. It does not reproduce the world.

If the badge vendor's original export was missing the last two hours of day three, a perfect rebuild reproduces an attendance figure that is wrong, and reproduces it with total confidence every time. Immutability protects the input from you. It offers nothing against an input that was defective when it arrived, and it can make a defective input harder to question, because the number is now stable and stability reads as correctness.

There is also a class of source you cannot make replayable at any price. An API that returns current state with no history, a partner feed that overwrites a shared file weekly, a survey platform that purges responses after 18 months. For those, the raw layer holds your snapshots and the honest statement is that replay reproduces what you captured on the day, with the capture date attached.

Start by picking the closed edition you have been asked about most often, rebuilding it into a scratch schema, and diffing the four or five headline measures against what was published. If they match, you have learned that your pipeline is replayable for at least one edition, which is more than most teams can say. If they do not, you have found a bug that has been in your reporting for a year, and you found it on a Tuesday instead of in a board meeting.