Skip to content

Deciding when to cache AI output and when to regenerate it

AI for organisersUpdated 2026-08-238 min read

In short

Cache AI output against a hash of the exact aggregates it was written from, so the text is reused while those numbers hold and regenerated the moment any of them changes. That removes most of the spend, and it removes the drift where two people quote different figures for the same show.

Two people open the same exhibitor's post-show summary on the same afternoon. One reads that scans were up 18 per cent. The other reads that scans were up almost a fifth. In a meeting a week later they disagree about what the system said, and both of them are quoting it accurately.

Nothing was wrong with the data. The summary regenerated on each page view, the model phrased the same division two different ways, and there is no stored artefact anyone can point at. Deciding when to cache AI output solves that problem and a cost problem at the same time, and the second one is how you get the budget to fix the first.

Key the cache on the numbers themselves

The instinct is a time to live. Cache for five minutes, or an hour, or until midnight. It is easy to implement and it is a guess in both directions: wasteful when nothing has changed for a fortnight, and wrong when a data reload landed ninety seconds into the window.

The better key is a hash of the exact values the output was generated from. Take the aggregates that went into the prompt, serialise them in a fixed order, hash them, and store the generated text against that hash.

Now staleness is structurally impossible. Any change to any input value produces a different hash, which misses the cache, which regenerates. No expiry to tune, no reload hook to remember, no window during which the text and the numbers disagree.

Two more components belong in the key. The prompt version, so an edit to the instructions invalidates every stored output that used the old wording. The model identifier including its version, so an upgrade regenerates rather than leaving a portfolio half written by one model and half by another. Both are cheap to include and both prevent a class of confusion that is genuinely hard to debug six weeks later.

A worked example: 900 exhibitor summaries and one data reload

Take the post-show summary run for a show with 900 exhibitors.

The first run generates all 900. Using the figures from the cost arithmetic in Q24, at roughly 3.9 cents an answer that is about 35 dollars of model spend, and at ninety seconds of review each it is 22.5 hours of somebody's attention. Both of those are one-off costs for a real piece of work.

Then the operations team reloads badge scan data on the Wednesday after the show, because a scanner in hall 3 had been uploading late and 4,100 scans arrived after the first extract ran.

Without a cache, somebody reruns the report and regenerates 900 summaries, spending another 35 dollars and another 22.5 review hours to produce 860 texts that say exactly what they said before.

With an aggregate hash, the reload changes the scan totals for the 40 exhibitors in hall 3 and nothing else. Forty hashes miss, 860 hit. Model spend is 40 times 3.9 cents, about 1.56 dollars. Review is 40 items at ninety seconds, one hour. The saving on the second run is 33 dollars of tokens and 21.5 hours of a salaried person, and the second of those is worth roughly 970 dollars at a loaded rate of 45 an hour.

The tokens are the small half. The review is the large half, which is the same conclusion the cost arithmetic in Q24 reaches from the other direction, and it is the reason cache design is worth a designer's attention rather than being left as an implementation detail.

Extend that across a portfolio and the shape becomes clearer. Eight shows a year, 900 exhibitors each, and a habit of rerunning the post-show report three times per show as late data trickles in, gives 21,600 generations a year without a cache. With one, the first run per show is a full 7,200 and the two later runs regenerate only what moved, which on the hall 3 pattern is around 40 to 90 each. Call it 8,200 generations against 21,600, a hit rate above 60 per cent on a conservative reading, and roughly 300 review hours that nobody has to spend rereading unchanged paragraphs.

That figure is worth putting in front of whoever approves the work, because it is the argument that gets a cache built. The argument that gets it built properly is the one about two colleagues quoting different numbers, and that one is harder to price and matters more.

What should never be cached?

Three categories, and the boundaries are worth drawing before someone caches something they should not.

Anything whose inputs you cannot enumerate. If a generation reads from a source you do not hash, the key is incomplete and the cache will serve text that contradicts a number that moved. When in doubt, either bring the source into the key or leave the output uncached.

Anything personalised at read time. A summary that adapts to who is reading it has the reader in its input set, so the reader belongs in the key, and at that point you are caching per person and the hit rate collapses. Usually the right move is to cache the shared body and compose the personal part outside it.

Anything the reader expects to be live. During show week an operations lead asking about the last hour wants the last hour. The technique still applies, because the hourly aggregate has its own hash and it changes every time the aggregate refreshes, but the practical cache window shrinks to the refresh interval. Q25 covers how that interacts with response time targets on a show floor.

There is a fourth case that looks like a category and is really a bug. Teams sometimes exclude a surface from caching because its output "keeps changing" without asking why. Two runs over identical inputs producing different text at temperature zero means something in the input set is moving that nobody has identified, commonly a timestamp injected into the prompt, a row order that is not deterministic, or a retrieval step returning passages in a different sequence. Hashing the inputs surfaces that immediately, because the hash changes and you can see which value moved. Chasing it down is usually a twenty minute job that also fixes a reproducibility problem in your evaluation runs.

Why is the version history worth more than the saving?

A cache keyed on inputs gives you something more valuable than the saving, which is a record.

Each entry holds the generated text, the hash, the input values themselves, the prompt version, the model identifier, and the time it was created. That is a complete answer to the question somebody will ask in November about why the September report said what it said. Not an approximate answer reconstructed from logs. The actual text, alongside the actual numbers it was written from.

NIST's AI Risk Management Framework, published as NIST AI 100-1 in January 2023, puts the reason for keeping that record more crisply than I can. Its section on accountable and transparent AI opens: "Trustworthy AI depends upon accountability. Accountability presupposes transparency." A cache entry containing the inputs, the output and the versions is transparency in the only form that survives a staff change.

There is a regulatory echo of the same idea. Article 12 of the EU AI Act, Regulation (EU) 2024/1689, requires that high-risk systems "technically allow for the automatic recording of events (logs) over the lifetime of the system", with those logs supporting post-market monitoring and the deployer's own monitoring of operation. Whether your reporting stack falls inside that scope is Q36's question. Building the record anyway costs one extra table.

Regeneration is a change, so treat it as one

Once outputs are cached, a regeneration becomes an event with a before and an after, and that is worth using.

When 40 summaries regenerate after a data reload, the system knows which 40 and it knows both versions. Present the change as a diff against the stored text, so a reviewer reads what moved instead of rereading a paragraph. That turns a second review pass into a much smaller job and it is the pattern the approval screen work in Q21 depends on.

It also gives you a monitoring signal that costs nothing. Count regenerations per run and group them by cause: input changed, prompt version changed, model changed. A run where 700 of 900 summaries regenerate because an input moved is telling you something about your data pipeline that no dashboard is currently saying out loud. A run where the same 12 exhibitors regenerate every night is telling you one upstream job is nondeterministic.

Where this stops

Hashing inputs guarantees the text matches the numbers it was made from. It guarantees nothing about whether those numbers are right, and it will preserve a wrong figure with perfect fidelity for as long as the wrong figure sits in the warehouse.

There is a subtler failure worth watching for. A cache makes text durable, and durable text gets quoted. A sentence generated in a hurry on the Friday after a show, cached, and then pasted into a board pack in November has acquired an authority it did not earn, and the fact that it is reproducible makes it feel more checked than it is. Reproducible and correct are different properties, and a cache only supplies the first.

The last limit is operational. Hashing floating point aggregates is fragile, because a value that recomputes as 17.960000000000001 rather than 17.96 misses the cache for no reason anyone can see. Round to the precision you actually publish before hashing, and store the rounded values, so the key and the prose agree about what the number is.

The step this week is a count. Take one AI-generated surface in your stack and log, for a fortnight, how many generations it performs and how many of those produced text identical to what it produced the previous run. If the identical share is above about two thirds, you have a cache-shaped saving sitting in plain sight, and the numbers to justify building it. The broader set of decisions around running AI over event data sits with the pillar.

Questions people ask about when to cache ai output

What should the cache key be?
A hash of the values the output was generated from, plus the prompt version and the model identifier. Hashing the aggregates makes staleness impossible by construction, because any change to a number changes the key. Including the prompt version and model means a prompt edit or a model upgrade invalidates everything without anyone having to remember to clear the cache.
Why not just cache with a time to live?
A time to live guesses. Five minutes is wasteful when nothing changed and wrong when a data reload landed at minute two. Hashing the inputs replaces the guess with a fact, and time to live becomes a safety net for cases where you cannot enumerate every input rather than the primary mechanism.
How does caching stop two people quoting different figures?
Regeneration produces different wording and sometimes different rounding from the same numbers. If the summary regenerates on every page view, two colleagues open the same exhibitor and see two texts. Serving one stored text for one set of inputs means everybody reading that exhibitor this week reads the same sentence.

Related reading

All ai for organisers articles