Hallucination rate measurement for an assistant that answers with numbers

The claim came up halfway through a vendor review: our assistant is 97 per cent accurate. Somebody had graded thirty of its answers over a coffee, found one wrong, and 29 out of 30 is 96.7 per cent, which rounds up well.

Hallucination rate measurement is worth doing properly, and that figure carries almost no information. Thirty answers, picked by whoever happened to be looking, graded against a standard nobody wrote down. Run the interval on one error in thirty and the true error rate sits somewhere between 0.6 and 16.7 per cent, so the honest version of the slide reads: the assistant is between fairly good and nearly perfect, and we cannot yet tell which.

Turning that into a number worth quoting starts with the denominator, because every decision downstream inherits it.

What exactly are you dividing by?

The same grading pass over the same output produces very different rates depending on the unit, and almost nobody states the unit.

Take 200 answers from one show cycle, each generated from a pack of aggregates. Grade every answer as a whole: an answer is unsupported if any figure in it cannot be traced to a supplied value. Suppose 17 fail. That is 17 divided by 200, or 8.5 per cent.

Now count the numeric tokens instead. Those 200 answers contain 1,340 numbers, and 24 of them are the ones that caused the 17 failures, because some answers went wrong twice. 24 divided by 1,340 is 1.79 per cent, call it 1.8.

Same grading, same defects, and the two rates are 4.7 times apart. Both are correct arithmetic and they answer different questions. The answer-level rate is what a reader experiences, because a board paper with one invented figure in it is a bad board paper regardless of how many good figures surround it. The token-level rate is the one that moves when you change a prompt, and it is the more sensitive instrument for engineering work.

Report both, each with its denominator written next to it. A rate with no denominator attached is a slogan.

Fixing the sample before anybody looks at it

The sample has to come from the question log. Questions written from memory in a planning session are a different distribution from questions typed into the box on the Tuesday of build-up, and grading the first tells you little about the second.

Define the frame explicitly: every question put to the assistant between two dates covering one show cycle, excluding nothing. Then draw. If your traffic is lumpy, and event traffic always is, stratify by question type and record the sampling weight per stratum so you can reconstruct an overall rate afterwards.

Then freeze the sample. The failure this prevents is the one everybody commits by accident, which is grading until the number looks acceptable and stopping there. Once the 200 are drawn, all 200 get graded, including the ones that turn out to be tedious.

There is a good precedent for keeping part of it sealed. Jacovi and colleagues at Google DeepMind released the FACTS Grounding benchmark in January 2025 with 1,719 examples split into 860 public and 859 held back, and they were explicit about why: the leaderboard "contains both public and private splits to allow for external participation while guarding the integrity of the leaderboard". The internal version of that is cheap. Grade two thirds while you are tuning prompts, keep a third sealed, and open it only when you want a number that will leave the team. A set you have optimised against has stopped measuring you.

Marking one answer supported or unsupported

Grading is binary and the rubric goes in writing before the first answer is opened.

Two stages, in order. First, does the answer address the question that was asked. An answer that describes exhibitor mix when the question was about registration pacing gets disqualified at that stage and never reaches the accuracy test, because a fluent irrelevance scoring as supported would flatter the system badly. Second, is every claim in the surviving answer carried by the supplied aggregates. The two stage structure and how to score one note against a pack is Q4's subject, and the rubric there is the one to reuse.

Write down the awkward cases in advance, because they arrive within the first twenty answers and deciding them mid-pass is how a grading exercise loses its integrity. A correct value attached to the wrong period. A hedge that gives a range where the pack holds a point value. A refusal. A question the assistant answered correctly and incompletely. Each of those needs a rule, and any rule is better than a judgement call made twice differently.

Who does the marking is its own problem. A model grader makes 200 items affordable, and it brings biases of its own, which is why FACTS Grounding averages three judge models and says plainly that the reason is "to mitigate any potential bias of a judge giving higher scores to the responses produced by a member of its own model family". Aggregating judges blunts that and does not remove it. How far a model judge can be trusted, and the regrade that tells you is Q11's territory, and the short version is that a human has to regrade a random subsample or the whole rate is unaudited.

Why does a rate need an interval?

Because 17 out of 200 is a draw from a process, and the next 200 will not give 17.

The reflex is the interval taught in every introductory course: the observed proportion plus or minus 1.96 standard errors. For 8.5 per cent on 200 items the standard error is the square root of 0.085 times 0.915 divided by 200, which is the square root of 0.000389, or 0.0197. Multiply by 1.96 and you get 3.9 points, so the Wald interval runs from 4.6 to 12.4 per cent.

That interval is worse than it looks. Brown, Cai and DasGupta published a careful examination of exactly this calculation in Statistical Science in 2001, volume 16, pages 101 to 133, and their conclusion is blunt: "the chaotic coverage properties of the Wald interval are far more persistent" than the literature had appreciated, and common textbook prescriptions about when it is safe "are misleading and defective in several respects and cannot be trusted." They recommend the Wilson interval or the equal-tailed Jeffreys interval for small samples and the Agresti and Coull interval for larger ones.

The Wilson calculation is worth doing once by hand. With p at 0.085, n at 200 and z at 1.96, z squared is 3.8416. The centre moves from the observed 0.085 to 0.085 plus 3.8416 divided by 400, all divided by 1 plus 3.8416 divided by 200, which is 0.0946 over 1.0192, or 0.0928. The half width is 1.96 divided by 1.0192, times the square root of 0.000389 plus 0.000024, which is 1.9231 times 0.0203, or 0.0390.

So the Wilson interval runs from 5.4 to 13.2 per cent. Almost exactly the same width as the Wald version and shifted upward by about eight tenths of a point, which is the correction that earns its keep at low rates, where the Wald interval drifts toward zero and eventually crosses it.

Just under eight points of width on 200 graded answers. That is the real precision of the exercise, and quoting 8.5 per cent without it invites a comparison against next quarter's 7.9 per cent that the data cannot support.

How many answers do you need before an improvement is real?

Work it backwards from the difference you want to see.

Suppose you want to establish that a prompt change took the rate from 8.5 to 6.5 per cent. For a two-sample comparison at 95 per cent confidence with ordinary power, the required size per group is roughly 7.84 times the sum of p times one minus p across both groups, divided by the squared difference. That is 7.84 times the sum of 0.0778 and 0.0608, or 1.0866, divided by 0.0004, which comes to about 2,716. Call it 2,700 graded answers for the old version and 2,700 for the new one.

Nobody is grading 5,400 answers. Two responses to that are honest and one is not.

The honest ones: raise the bar, and only claim differences your sample can actually resolve, which at 200 items means several points; or switch to a paired design, running the same fixed question set through both versions and counting only the questions whose grade changed. Pairing removes the variance that comes from some questions being harder than others, and the arithmetic then runs on the count of disagreements instead of on two independent proportions, which needs far fewer items for the same resolution. That is the strongest practical argument for keeping a fixed question set at all.

The dishonest response is to grade 200, observe 8.5 against 6.5, and write "a 24 per cent reduction in hallucinations" in the release note.

Where this stops

Your rate is conditional on your corpus, your rubric and your graders, and it does not travel. Vectara has published a hallucination leaderboard since 2023 that scores models on summarising a supplied passage over one fixed article corpus, which makes it a good instrument for ranking models against each other and a poor predictor of what any particular model will do against your registration schema. Borrowing a leaderboard figure as your expected rate is a category error.

The rate is also silent about severity. An assistant that gets a session count wrong and an assistant that gets the rebooking rate wrong both score one unsupported answer, and only one of those ends up in a renewal forecast. If you weight anything, weight by where the answer goes, and keep the unweighted rate alongside so the weighting stays visible.

The deepest limit is the definition. What you have measured is the proportion of answers unsupported by the aggregates you supplied, which is a real and useful thing, and it is not the proportion of answers that are true. If the pack itself carries a fabricated figure, every answer quoting it grades as supported. The routes by which a number with nothing behind it reaches a report sit upstream of every rate on this page, and they are Q7's subject.

Take the last 40 answers your assistant produced, write one page of grading rules before you open any of them, mark each supported or unsupported, and compute the Wilson interval. If the interval is wider than the difference you were hoping to report about your AI layer, you now know how many more you need to grade, which is the only useful thing a first pass can tell you.