Writing an AI summary of show metrics that survives a line by line check

The paragraph took nine seconds to generate and read like something a good analyst would write. Fourteen aggregates went in, one paragraph came out, and the operations lead skimmed it and forwarded it.

One number in it was 9,125. The supplied value was 9,215. Two digits transposed, in a session scan total nobody has memorised, in a sentence that was otherwise correct. An AI summary of show metrics is worth generating, and it is worth generating in a way where that transposition is caught by a machine in the same second it is written, because no human reader is going to catch it.

The input contract, and nothing outside it

Here is the whole input for a post-show summary. Fourteen rows, each with a name, a value and a period.

registrations_total 18,640 and registrations_total_prior 17,905. verified_attendance 13,908 and verified_attendance_prior 13,412. attendance_rate 0.746 and attendance_rate_prior 0.749. exhibitor_count 612 and exhibitor_count_prior 588. net_square_metres_sold 21,470 and net_square_metres_sold_prior 20,910. international_share 0.213 and international_share_prior 0.197. sessions_delivered 84 and session_attendance_scans 9,215.

Two properties of that list matter more than its contents. It is closed, so anything in the output that is not in the list is a defect by definition. And it contains the derived figures already, so attendance_rate arrives as 0.746 instead of being something the model works out.

Everything else people are tempted to include should stay out. No rendered dashboard. No screenshot. No table of eight thousand rows with an instruction to find the interesting parts. A list of fourteen named values is small enough that a person can hold all of it, which is the same property that makes the checking step trivial.

Why forbid the model from doing arithmetic?

Because the error is silent and the benefit is zero.

Ask a model to divide 13,908 by 18,640 and it will usually return 74.6 per cent. Occasionally it will return 74.9, or 76.4, and nothing in the sentence signals which case you are in. The percentage sits there looking exactly like a percentage. Meanwhile the same division in SQL is free, exact, and already needed for the dashboard, so the model is being handed a job that was done ten minutes earlier by something that cannot get it wrong.

I would go further than the usual advice here. Precompute every figure the paragraph could plausibly want, including the ones you think it will not use: the differences, the percentage changes, the shares, the per-day averages. A pack of fourteen becomes a pack of twenty-two, the prompt is still short, and the model's remaining job is selection and phrasing, which is what it is actually good at. Any figure it might want and cannot find is a figure it might invent.

There is a real cost. Prose built entirely from precomputed values sometimes reads a little flat, because the model cannot reach for the comparison that would have made the sentence land. In a document that decides next year's hall allocation, flat is fine.

Matching every figure back by string comparison

Now the check, which is the part most teams skip and the part that makes the rest work.

Parse every numeric token out of the generated paragraph. For the summary above that gives 18,640, 17,905, 13,908, 13,412, 74.6, 74.9, 612, 21,470, 588, 20,910, 21.3, 19.7, 84 and 9,125. Fourteen numbers, ignoring the year labels 2026 and 2025.

For each token, look for a supplied value that matches after normalisation. Normalisation needs three rules and no more: strip thousands separators, so 18,640 matches 18640. Treat a proportion row as also matching its percentage form, so attendance_rate 0.746 matches 74.6. And apply a rounding tolerance to one decimal place, so 21.3 matches 0.213 and 74.9 matches 0.749.

Thirteen of the fourteen tokens find a match. The fourteenth, 9,125, does not, because the supplied value was 9,215. Thirteen out of fourteen is 92.9 per cent at the token level, and at the document level it is a fail, because a summary going to a board with one wrong figure in it is a wrong summary. Both numbers are worth logging. The percentage tells you whether the prompt is drifting week to week. The pass or fail decides whether this document leaves.

The whole check is about thirty lines of code and runs in single-digit milliseconds. It catches transpositions, dropped digits, invented figures and any number the model reached for out of its own training data. It does not catch a correctly quoted number attached to the wrong label, which is the recompute-and-compare job Q9 handles.

What hallucination rate should you expect on this task?

Vectara has published a hallucination leaderboard since 2023, and its method is the closest public analogue to what a show metrics summary asks a model to do. Models are given a document and the instruction "Summarize using only the information in the given passage. Do not infer. Do not use your internal knowledge." Generation runs at temperature 0, summaries are capped at 20 per cent of the source length, and the results are scored by Vectara's own hallucination evaluation model over a corpus of more than 7,700 articles ranging from 50 to 24,000 words across news, technology, science, medicine, legal, sports, business and education.

On the 11 May 2026 update, the measured rates run from 1.8 per cent at the best to 24.2 per cent at the worst. That spread is the useful part. Model choice moves this figure by more than an order of magnitude on an identical task with an identical instruction, so the decision of which model writes your exhibitor reports is a measurable one with a large effect.

Two caveats before anyone transplants those numbers. A fourteen-row aggregate pack is a much easier input than a 24,000-word article, so a well-built metrics summary should land well below the leaderboard figures for the same model. And the residual errors on a metrics task are disproportionately numeric, because numbers are where a summary has the least redundancy: a wrong adjective is recoverable from context and a wrong digit is not.

Decomposing the paragraph into claims

The string check verifies figures. It says nothing about the sentences around them, and a summary can pass every numeric check while asserting something the pack does not support.

Min and colleagues published FActScore at EMNLP 2023 as a way of handling exactly this. Rather than judging a passage as a whole, it breaks the generation into atomic facts and computes the percentage supported by a reliable knowledge source. Their human evaluation found ChatGPT scoring 58 per cent on biography generation, and their automated estimator reproduced human scores with under a 2 per cent error rate, which is the finding that makes the approach practical at volume.

For a show metrics summary the atomic facts are short and the knowledge source is your fourteen rows, so this is easier than the paper's setting. Split "612 exhibitors occupied 21,470 net square metres, up from 588 exhibitors and 20,910 square metres" into four claims and check each against a row. The claim that usually fails is the one carrying a causal or directional word the pack never established, and those are worth counting separately from the numeric failures, because they need a different fix.

Letting the model say nothing

The last piece of the instruction is permission to omit.

If session_attendance_scans is missing from the pack this week because the sensor feed failed, a model told to write a complete summary will write about sessions anyway, using whatever the pack does contain and some plausible framing. Told explicitly that any metric absent from the list must be left out of the summary entirely, it produces a shorter paragraph, which is the correct output.

Test for it directly. Delete two rows from the pack, regenerate, and check that the two metrics disappear from the prose. If they survive in any form, the instruction is not working and no amount of downstream checking will fix a model that fills gaps.

Where this stops

Everything above makes the paragraph faithful to fourteen numbers. Whether those fourteen numbers describe the show is a separate question with a separate answer.

The check has a specific blind spot worth naming: a figure that is wrong in the pack passes every test, because the pack is the ground truth by construction. If verified_attendance was computed from a scan table missing an entire hall, the summary will quote it, the string match will pass, and the error is now in prose. The ways a figure with nothing behind it reaches a report are Q7's subject, and they matter more than the summarisation step.

The second limit is that a 92.9 per cent token match rate on one paragraph is not a measurement of anything. One document, fourteen tokens. Treating the rate as a real number needs a sample, a denominator and an interval, and scoring whether the surrounding claims hold up needs the two stage grounding check in Q4.

Take your most recent show summary, list every number in it, and try to match each one against a value your warehouse actually holds. The count of numbers you cannot match is the size of the gap in your reporting layer, and it takes about twenty minutes on a single document.