Offline evaluation versus live monitoring for an assistant in daily use

Your assistant scores 88 per cent on the evaluation set. During the September show it answers 640 questions and three of them end up in a complaint thread. Both facts are true and they are measuring different things, which is the whole of the offline evaluation versus live monitoring problem and the reason teams keep arguing past each other about whether the feature works.

An eval set tells you what happens on questions somebody sat down and wrote. A log tells you what people ask at 08:40 on day one with a hall full of exhibitors. The overlap is smaller than anyone expects.

What an offline eval set is genuinely good for

Comparability. A frozen question set with known answers is the only instrument that lets you say version B is better than version A, because everything else about the comparison is held still.

That is the design principle behind HELM, which Liang and colleagues at Stanford's Center for Research on Foundation Models released in 2022. Their contribution was coverage and standardisation across 42 scenarios, 16 core and 26 targeted, evaluated over 30 models against 7 metrics reported side by side, among them accuracy, calibration and efficiency. The number worth carrying into your own work is what they found about the state of evaluation before they started: across the 30 models, average coverage of the core scenarios was 17.9 per cent, and their run took it to 96.0 per cent.

Seventeen point nine per cent. That is what "we evaluate our model" meant in practice for a field that evaluates for a living, and it is a fair description of most internal eval sets I have seen. Everyone has one. Almost nobody knows what fraction of real behaviour it touches.

The other thing a frozen set gives you is a place to put a bug. When a show director reports a wrong answer, the fix is not finished until that exact question, with its verified correct result, is in the set. Do that consistently for a year and the eval set becomes a record of everything that has ever gone wrong, which is worth considerably more than a set of questions somebody imagined in a planning session.

What does live traffic tell you that an eval set cannot?

Which questions exist.

Koh and colleagues made the general point with WILDS in 2021, a benchmark of 10 datasets built specifically around shifts that occur in deployment, such as moving between hospitals or between camera traps. Their finding was that standard training gives substantially lower out-of-distribution performance than in-distribution performance, and that the gap survived the methods designed to close it.

Your eval set is the in-distribution sample by construction, because you wrote it. Live traffic is the out-of-distribution one. The shift is not subtle in event work: an eval set written in June contains clean questions about registration pacing, and a log from show week contains questions about a specific hall on a specific afternoon, half-typed, with a stand number in them, asked by someone who has been on their feet since six.

Run the coverage check rather than assuming. Take a week of real questions, 640 of them, and map each to a category in your eval set. Suppose 380 map to something. That is 59.4 per cent covered and 260 questions, 40.6 per cent, that your offline score says nothing whatsoever about.

Sampling twenty answers a week, and what that buys

The standard live monitoring routine is to log everything and grade a weekly sample by hand. It is a good routine and the sample size is usually wrong.

Take 20 answers from the 640 and grade them. Suppose 4 fail. The point estimate for the failure rate is 20 per cent, which across 640 answers implies 128 failures. Now the interval. The standard error is the square root of 0.2 times 0.8 divided by 20, which is the square root of 0.008, or 0.0894. Two standard errors is 0.179, so the interval runs from about 2 per cent to about 38 per cent, or between 13 and 243 failures across the week.

Thirteen failures is a good week. Two hundred and forty-three is an emergency. A sample of 20 cannot distinguish them, and reporting "20 per cent failure rate" from that sample gives a committee a number with no information in it.

To get the interval to plus or minus 8 points you need two times the square root of 0.16 over n to equal 0.08, so 0.64 over n is 0.0016 and n is 400. That is too many to grade by hand every week, which is the honest reason model grading exists at this stage, with the caveats that come with trusting a model judge.

How should you spend a fixed grading budget?

Stratify, and stop sampling uniformly.

Split the week's questions into strata before you sample. A workable split for an event assistant is questions that returned no rows, questions whose answer contained a percentage, questions containing a stand or badge identifier, and everything else. Then sample within each.

The reason is that the failure rates differ enormously between those groups and the volumes do not. Empty result sets might be 3 per cent of traffic and a large share of your complaints. Sampling 20 uniformly from 640 will pick up roughly one of them. Sampling 10 from that stratum alone tells you something.

Report the strata separately and weight them back to a total only if someone insists. The weighted total is usually the least useful number in the report, because it averages a rare expensive failure with a common harmless one.

What the log has to contain to be worth sampling

A log of prompts and completions is the version most teams build and it is close to useless a fortnight later, because you cannot reconstruct why the answer was what it was.

• The question text as typed, including the typos and the abandoned first attempt, since a reformulated question is itself a failure signal.
• The resolved query or the named aggregates handed to the model, which is what you grade against.
• The retrieved document identifiers, where documents were involved.
• The model version and prompt template hash, so a week of results can be attributed to a configuration.
• What the user did next, at minimum whether they asked a follow-up within two minutes or left.

That last field is the one nobody logs and the one with the most signal in it. A question followed 30 seconds later by the same question rephrased is a failed answer that no grader flagged, and you can count those automatically. In a week of 640 questions, if 74 are followed by a near-duplicate inside two minutes, that is 11.6 per cent of traffic telling you something went wrong, measured with no human grading at all.

Keep a retention decision attached to the log too, because it holds attendee-identifying text that people typed into a search box. Grading needs recent data and does not need three years of it.

Feeding the log back into the offline set

The two exercises are meant to be a loop and most teams run them as two disconnected rituals.

Once a quarter, pull the distinct question shapes from the log, count them, and compare against the eval set. Every uncovered shape asked more than a handful of times becomes a new eval question, with its correct answer verified by hand against the warehouse. Every eval question nobody has asked in six months gets a decision: keep it because it guards something important, or retire it.

That decision matters more than it sounds. An eval set grows monotonically if nobody prunes it, and a suite of 300 questions that takes an hour to run gets run less often than a suite of 60 that takes ten minutes. The suite you actually run before a change beats the comprehensive one you run quarterly. Holding the suite stable between rebuilds is what makes version comparison possible in the first place, which is the discipline in regression testing when the model changes underneath.

Where a live failure turns out to be a retrieval failure, the eval question you add should be scored against the retrieved documents as well as the final answer, because otherwise you have added a test that goes green when the model guesses correctly from the wrong source. That split has its own method in measuring retrieval quality separately.

The limit

Both instruments measure whether the answer matched an expectation. Neither measures whether anyone acted on it, and for an assistant that exists to change what a show director does on a Tuesday, that is the outcome you actually care about.

There is also a floor on what live monitoring can see. Questions people stopped asking because the assistant answered them badly in March never appear in the log, and the log will look cleaner every month as the audience narrows to the people the tool happens to serve. Rising accuracy alongside falling volume is the pattern to watch for, and it reads as success on every dashboard I have seen. Track distinct users and distinct question shapes per week alongside the accuracy figure, on the same chart, or the narrowing is invisible.

The wider set of controls around a deployed assistant sits on the AI platform side.

Start with the coverage check. Take last week's log, whatever size it is, and map each question to a category in your eval set. If more than a quarter of the questions map to nothing, your offline score is describing a product other than the one people are using, and rebuilding the set is worth more this month than any change to the model.