Grounding AI summaries in aggregates the reader can check themselves

The pre-show pacing note arrived on Monday at 07:00, six sentences long, and five of them were fine. The sixth said marketing attributes the slowdown to the later campaign launch.

Nobody had told the system anything about campaign launch dates. It had been handed nine numbers and had written a sixth sentence that sounded like the kind of thing a pacing note says. Grounding AI summaries in aggregates is the discipline that catches that sentence, and the useful version of it has two separate stages rather than one general check for truthfulness.

Why does a grounding check need two stages?

Google DeepMind and collaborators published the FACTS Grounding benchmark in January 2025, and its structure is worth copying wholesale because it solves a problem most internal checks walk straight into.

Stage one asks whether the response fulfils the user's request. Stage two asks whether the response is fully grounded in the supplied document. Their reason for separating them is that a factuality score on its own can be gamed by saying less. A model that answers a request for a pacing note with the single sentence "Registrations stand at 11,420" is perfectly grounded and completely useless, and it scores 100 per cent on any check that only looks at whether claims are supported. FACTS disqualifies responses that fail the first stage and treats them as inaccurate, and the paper reports that doing so reduces the final factuality score by 1 to 5 per cent across the models evaluated.

That 1 to 5 per cent is the size of the hole in a one-stage check. It is small enough that nobody notices it and large enough to reorder a leaderboard, which the paper also records happening.

For an event reporting note the first stage is easy to write and easy to forget. The request was a pacing read at day minus 45 covering volume, week-on-week movement and channel mix. A response that covers volume and skips the other two is ineligible before anyone examines whether its numbers are right.

The aggregate pack behind one pacing note

The material supplied to the model is nine rows, each with a name, a value, a period and a source table.

registrations_cumulative_current, 11,420, 2026 edition at day minus 45, fact_registration. registrations_cumulative_prior, 12,180, 2025 edition at day minus 45, fact_registration. registrations_last_7_days, 1,046, 2026 edition, fact_registration. registrations_last_7_days_prior, 1,310, 2025 edition, fact_registration. international_share_current, 0.213, 2026 edition, fact_registration. international_share_prior, 0.197, 2025 edition, fact_registration. allotment_registrations_current, 2,104, 2026 edition, fact_registration. allotment_registrations_prior, 1,488, 2025 edition, fact_registration. days_to_open, 45.

Nine rows is small on purpose. FACTS Grounding evaluates documents up to 32,000 tokens, and long-context grounding is a genuinely hard task. A pacing note does not need it. Every row in that pack is a number somebody can open in a report and see, which is the property that makes the whole check possible.

Notice what is absent. There is no row for campaign spend, no row for competitor dates, no row for last year's weather. Those things might all be relevant to why pacing is soft, and none of them is available to the model, so a sentence about any of them is unsupported by construction. Deciding the contents of the pack is therefore also deciding what the note is allowed to say, and that decision belongs to whoever owns the report.

Scoring the note sentence by sentence

Split the generated note into sentences and put each one next to the row or rows it needs.

Sentence one, registration for the 2026 edition stands at 11,420 with 45 days to open, depends on rows one and nine. Supported.

Sentence two, that is 6.2 per cent behind the 2025 edition at the same distance, which held 12,180, depends on rows one and two plus one division. 12,180 minus 11,420 is 760, and 760 divided by 12,180 is 0.0624, so 6.2 per cent is right. Supported.

Sentence three, the last seven days added 1,046 registrations against 1,310 in the comparable week last edition, depends on rows three and four. Supported.

Sentence four, international share has improved to 21.3 per cent from 19.7 per cent, depends on rows five and six with a decimal shift. Supported.

Sentence five, exhibitor allotment registrations are running 41 per cent ahead at 2,104, depends on rows seven and eight. 2,104 minus 1,488 is 616, and 616 divided by 1,488 is 0.414. Supported.

Sentence six, marketing attributes the slowdown to the later campaign launch, depends on nothing in the pack. Unsupported.

Five of six sentences trace, which is 83.3 per cent. Under the FACTS response-level rule the note scores zero, because a response counts as accurate only when every claim in it is grounded and a single ungrounded claim that carries information marks the whole thing inaccurate. Both numbers are worth keeping. The 83.3 per cent tells you how the prompt is behaving over time. The zero is what governs whether this particular note goes out.

Maynez, Narayan, Bohnet and McDonald called sentence six an extrinsic hallucination in their 2020 study of abstractive summarisation, meaning content that neither contradicts the source nor appears in it. They found hallucinations in more than 70 per cent of single-sentence summaries, and, in the finding that should worry anyone shipping this, over 90 per cent of the extrinsic ones were erroneous. Plausible additions are usually wrong additions, and the campaign launch sentence is exactly the shape of thing a reader forwards.

What counts as grounded when the sentence does arithmetic?

This is where teams disagree, and the disagreement is worth resolving explicitly rather than by default.

A strict reading says only values present in the pack may appear in the output, which forbids sentence two entirely, because 6.2 per cent is nowhere in the nine rows. A loose reading says any arithmetic over supplied rows counts as grounded, which permits the model to compute anything and puts you back in the position of trusting its arithmetic.

The position I would take sits between them. Permit arithmetic only when the operands appear in the same sentence, which is what sentence two and sentence five both do. A reader who can see 11,420, 12,180 and 6.2 per cent together can check the division in their head. A reader shown only 6.2 per cent cannot, and the model's error rate on that division is now invisible instead of obvious.

The cleaner alternative is to precompute the derived figures too, so the pack carries a pacing_index row and the model never divides anything. That is the input contract Q6 works through for a full show metrics summary, and it is the better default for anything that goes to exhibitors.

Who checks the checker

Running the grounding check with a language model raises the obvious question, and FACTS Grounding answers it in a way small teams can copy at reduced scale.

They use three judge models, Gemini 1.5 Pro, GPT-4o and Claude 3.5 Sonnet, and average the results, because models favour their own outputs. The paper quantifies that bias: a mean increase of 3.23 per cent when a model grades itself. They also ensemble the eligibility decision by consensus, so a response is only disqualified when all three judges agree it fails to answer the request, which keeps the filter conservative.

Three judges on every pacing note is not a sensible use of anyone's budget. Two rules get most of the benefit. Never let the model that wrote the summary be the model that grades it. And keep a small holdout of notes graded by a person, because the FACTS team selected their judge prompts by measuring agreement against human labels on a private set of 402 responses, and without some human-labelled anchor you have no idea whether your judge is any good.

Making the trace visible to the reader

The check above happens before publication. The part that changes behaviour happens after.

Render each sentence with the row names it used, either inline or on hover, so the reader who doubts the international share number can see that it came from international_share_current and open the underlying report. This is cheap, because the mapping already exists as a by-product of scoring, and it converts a summary from something people either believe or do not into something they can interrogate at the level of one sentence. Attaching the query, the table and the run timestamp to each figure so a disputed number can be settled in a minute is the fuller provenance treatment in Q5.

Where this stops

Grounding is a check on the writing step and nothing else. If registrations_cumulative_current is 11,420 because the query behind it dropped a registration type, the summary will be perfectly grounded, perfectly traceable and wrong, and the tracing will make it more persuasive.

The second limit is that sentence-level scoring rewards short notes. A model that writes three sentences scores better than one that writes eight, and the eight-sentence note is often the more useful document. The eligibility stage exists to push back against that, and it only pushes as hard as the request you wrote. Vague requests produce thin summaries that pass both stages.

The third is measurement. One note scored 83.3 per cent tells you almost nothing, because the denominator is six. Turning a grounding check into a rate anybody should quote needs a sample large enough to carry an interval, which is Q8's subject.

This week, take the last generated summary your team sent to anybody and write the row name next to each sentence in it. Count the sentences with nothing to write. That count, over the total, is your current grounding rate, and it is the first honest number you will have about the AI layer over your reporting.