Numeric consistency checks that catch a wrong figure before publication

The email arrived at 07:40 on the Tuesday after the show. "Your performance note says our stand cost 417,600 pounds. It cost 41,760. What else in here is wrong?"

That last question is the expensive part. One misplaced comma in a generated exhibitor note, and an account manager spends a week re-verifying every figure in every note that went out, by hand, for 584 exhibitors. Numeric consistency checks exist so that comma never leaves the building, and they work by recomputing each figure from the aggregate that produced it instead of trusting the text.

Matching a number and recomputing it answer different questions

There is a cheaper check that runs first, and it is worth being clear about where it stops.

The cheap version parses every numeric token out of the generated text and looks for the same value in the pack of aggregates the model was given. That catches invented figures, transposed digits and anything the model reached for out of its training data, it costs almost nothing, and supplying a closed pack and matching every token back against it is Q6's subject and the right first control.

It answers one question: does this token appear in the input. Recomputing answers a harder one: is this token the correct answer to the claim the sentence is making.

The gap between those two shows up in two places. The first is a correct value carrying the wrong label. If the pack holds an attendance rate of 74.6 per cent for this edition and 74.9 for the last one, and the text reports 74.9 for this edition, a string comparison finds 74.9 in the pack and passes. A recompute against the labelled period fails it immediately.

The second is the derived figure. A pure match-back forces you to precompute every number the text might want, because anything not in the pack is a defect by definition. That is a defensible position and it makes the prose flat, since the model can never state a ratio you did not anticipate. Once a recompute layer exists you can relax it: let the text derive a cost per lead, and verify the derivation. I would take that trade in an exhibitor-facing document, where the useful sentences are almost all ratios.

Parsing a number with its claim attached

The parsing job is harder than a regular expression over digits, and the reason is that a bare number is not checkable. You need the value, the unit, the entity it describes and the period it covers.

Zhao, Cohen and Webber took this seriously at Findings of EMNLP in 2020, pages 2237 to 2249, in work on reducing quantity hallucinations in abstractive summarisation. Their system, Herman, learns to recognise and verify what they call quantity entities, which they describe as dates, numbers and sums of money, then up-ranks the summaries in a beam whose quantity terms are supported by the source text. Their reported result is that the up-ranked summaries score higher on ROUGE precision without a comparable loss in recall.

Two things carry across to an event reporting pipeline. Quantity entities are the right unit of analysis, because dates, counts, areas and money each need a different comparison rule. And re-ranking is the wrong response for a document with your logo on it. Herman picks the best of several candidate summaries, which suits a research setting with no human downstream. A performance note going to a paying exhibitor should fail closed, hold the document, and put the mismatch in front of a person.

In practice the extraction produces a tuple per figure: the value as written, the normalised value, the metric name the surrounding clause implies, the entity, and the period. For the sentence "your 36 square metre stand generated 428 leads, up 18.6 per cent on last year", that is three tuples, and the third one carries a comparison across two periods, which is what makes it recomputable at all.

Nine numbers in an exhibitor note, one comma out of place

Here is the note that produced the Tuesday email, with the aggregates behind it.

The pack held stand area 36 square metres, rate 1,160 pounds per square metre, leads captured 428, leads captured at the prior edition 361, meetings booked 62 and meetings attended 51. Six values, each a row somebody can open.

The generated note quoted nine numbers. Six of them are direct: 36, 1,160, 428, 361, 62 and 51, each matching a pack row on lookup. Three are derived and get recomputed.

Total stand cost, quoted as 417,600 pounds. Recompute: 36 times 1,160 is 41,760. Mismatch by a factor of ten, and the shape of the error is a comma one place too far right.

Lead growth, quoted as 18.6 per cent. Recompute: 428 minus 361 is 67, divided by 361 is 0.18560, which rounds to 18.6. Pass.

Meeting attendance, quoted as 82.3 per cent. Recompute: 51 divided by 62 is 0.82258, which rounds to 82.3. Pass.

Three recomputes, one failure, and the failure is the only number in the note that a plain match-back would also have caught, since 417,600 appears nowhere in the pack. The recompute earns its place in what it hands the reviewer: the correct value, the two operands, and the arithmetic. A checker that says "figure 3 is unsupported" starts an investigation. A checker that says "417,600 should be 41,760, from 36 times 1,160" ends one.

The propagation is worth following too. The account manager's covering email divided the stand cost by 428 leads to give a cost per lead of 975.70 pounds, when the true figure is 41,760 divided by 428, or 97.57. One comma became a tenfold error in the single number an exhibitor uses to decide whether to rebook.

What tolerance should a numeric check use?

Set it at half a unit in the last digit the text actually quotes. If the text says 18.6 per cent, accept anything from 18.55 to 18.65 and reject the rest. If it says 19 per cent, accept 18.5 to 19.5. The tolerance follows the precision of the claim, which is the only defensible reading of what the sentence asserts.

Relative percentage tolerances are the common alternative and they behave badly on money. Allow one per cent on the cost per lead of 97.57 and you accept anything from 96.59 to 98.55. Multiply that band back across 428 leads and it spans 41,340 to 42,180 pounds, a width of 840 pounds on an invoice of 41,760. The check passes, the exhibitor's own arithmetic disagrees with yours, and the conversation is worse than if you had published nothing.

Currency and unit handling needs its own rules, decided once: square metres against square feet, thousands separators against decimal commas on any show with European exhibitors, percentages against proportions, and dates that determine which edition a figure belongs to. Every one of those has produced a tenfold or a twelve-month error somewhere, and every one is a few lines of normalisation code.

What should the checker do when a number fails?

Fail the document, log the mismatch, and regenerate at most once before escalating to a person.

Repairing the number in place is tempting and wrong. A pipeline that silently patches 417,600 into 41,760 has removed your only signal that the generation step is unreliable, and the next failure will be one the patcher cannot fix. Count the failures instead, and watch the count.

What the escalation shows matters as much as the fact of it. Kryscinski, McCann, Xiong and Socher built their factual consistency model at EMNLP in 2020, pages 9332 to 9346, and the design decision to copy is the third of their three training tasks: for each summary sentence judged inconsistent, extract the inconsistent span from it. A verdict on the document is nearly useless to a reviewer. A highlighted token with the recomputed value beside it is actionable in seconds.

Their training data generation is instructive in a second way. They built weak supervision by applying rule-based transformations to source sentences, among them a number swap that replaces one numeric value with another from the same document, with an example in the paper turning a 12.6 million dollar lien into a 3.45 million dollar one. That transformation is a fair description of the most common failure in generated event reporting, and it means you can build your own regression fixtures the same way: take a note that passed, swap one figure for another real figure from the same pack, and confirm your checker fails it.

Where the figure came out of a retrieved document rather than an aggregate, the recompute has nothing to run against and the question becomes whether the right passage was retrieved at all, which is scoring the retriever on its own in Q17. Es and colleagues built the Ragas framework around that separation.

Where this stops

A recompute verifies a figure against an aggregate. It cannot verify the aggregate. If the leads count of 428 was pulled from a scan table missing one of the two badge readers on the stand, every number in the note reconciles perfectly and the whole note is wrong. The checker is a consistency instrument and it makes no claim about truth.

The second limit is coverage, and it is worth measuring rather than assuming. Across a month of generated notes you might find 1,270 numeric tokens, of which 1,043 have either a pack lookup or a recompute rule. The other 227, 17.9 per cent of the total, are figures nobody wrote a rule for: a forecast quoted as a range, a date, a count lifted from free text feedback. Those go out unchecked, and the honest reporting of your checker's performance includes that fraction.

The third is that a recompute layer makes it safe for the model to state derived figures and does nothing about the model getting a calculation wrong in the first place, which happens for structural reasons worth understanding separately. Why a language model gets arithmetic wrong and what belongs in SQL is Q19's argument, and it sets the default: compute upstream, check downstream, and treat every remaining calculation in the text as something the checker has to earn.

Take the last generated document you sent to somebody outside your organisation. Count the numbers in it, then count how many you could recompute from a single aggregate in under a minute. The gap between those two counts is the part of your AI layer currently running unverified, and on a first pass it is usually larger than anyone expects.