Building an evaluation set for an event assistant from real questions

Somebody in the planning meeting volunteers to write some test questions. A week later there are thirty of them, neatly phrased, each one a complete sentence with correct capitalisation, and not one of them looks like anything a sales director types into a box at twenty past eight on the first morning of build-up.

The questions you need already exist. They are sitting in the channel where the commercial team asks your data person for numbers, and an evaluation set for an event assistant built out of that channel will describe your organisation's actual demand for information. Harvesting beats inventing, and the work is mostly clerical.

Where do the questions come from?

Four sources, in descending order of usefulness.

The request channel is the best one. Whatever you call it, there is a place where people ask for numbers and somebody answers. It has the real phrasing, the abbreviations, the half-sentences, and, more valuable than any of that, the follow-up messages where the asker says that is not what I meant.

The second is the ad hoc SQL your data person has already written. Every one of those queries was written in response to a question, and the query is the answer, so these items arrive half-graded.

Third, the assistant's own log if it is already running. Fourth, and only fourth, questions written from imagination, which are worth including where you know a capability matters and nobody has needed it yet.

One warning about the channel. It contains personal information, exhibitor commercial terms and occasionally somebody's salary. Strip it before it becomes a test fixture that gets copied into a prompt, because an evaluation set has a habit of ending up in more places than the data it came from was ever cleared for.

From 486 asks to 84 graded items

Here is the arithmetic on a single quarter, with numbers you can follow.

The channel holds 1,412 messages since April. Read them and 486 are questions about event data, which is 34.4 per cent; the rest are answers, thanks, and arrangements about who is picking up the badge stock. Cluster the 486 by what is being asked rather than by wording, and they collapse to 214 distinct intents. That collapse ratio is the first useful finding, because it says the same 214 things are being asked over and over, and that a well-built assistant has a smaller job than the volume suggests.

Now rank the 214 by how often they were asked. The head is short. The top 40 intents account for 298 of the 486 asks, or 61.3 per cent, and they are the questions you would have guessed: registration against the same point last year, rebooking count, top exhibitors by scans, international share.

The remaining 174 intents carry 188 asks between them, so almost every one was asked once. That tail is where an assistant fails, and it is where a wrong answer costs most, because the person asking a once-only question has no prior expectation to check the answer against.

So sample both. Take all 40 head intents, then draw 44 of the 174 tail intents at random, which is 25.3 per cent of the tail. The set is 84 items, weighted so you can report a head rate and a tail rate separately and still combine them.

Then price it before you promise it. Writing and checking the truth for one item takes roughly 25 minutes once the schema is familiar, so 84 items is about 2,100 minutes, or 35 hours. One person, one week, and that estimate is the thing to put in front of whoever is asking why the assistant has not shipped.

What makes one item gradeable?

Six fields per item, and the set is worth very little if any of them are missing.

• The question as asked, with the typo and the abbreviation left in.
• The resolved question, where you have decided what an ambiguous phrase means, or deliberately kept the ambiguity because handling it is part of the test.
• The SQL that produces truth, written by a person against the warehouse.
• The expected answer, stored as that query plus an as-of date so it stays live as the warehouse moves.
• The comparison rule, since some items want an exact row set, some a scalar within a tolerance, and some the correct top five in the correct order.
• The provenance, meaning who established the truth and when.

The fourth of those causes the most argument, so it is worth defending. Take a question about how many exhibitors from the last edition have rebooked for the next one. The true answer changes every day as contracts land, so freeze it as 412 and the item is wrong by Thursday. Store the query instead, re-run it at grading time, and the item stays correct for years, at the cost that its difficulty drifts as the underlying distribution changes. The alternative is a warehouse snapshot per evaluation run, which is cleaner and considerably more expensive to keep.

Include items whose correct answer is that no answer exists. Ask about attendee satisfaction for an edition where no survey ran, and the right behaviour is to say so. Roughly one item in five should have refusal as the correct grade, because a set where every item has an answer trains you to ship an assistant that always produces one.

Four criteria worth holding the set to

Bowman and Dahl set out four criteria for benchmarks in natural language understanding at NAACL in 2021, pages 4843 to 4855, and all four survive translation into a small internal eval set.

Validity. An item is invalid if a system can pass it without doing the work. Event data has an obvious version of this failure: year on year questions mostly have "up" as the true answer, so if 70 of your 84 items point upward, a system that says up every time scores 83.3 per cent while knowing nothing. Balance the direction of the truth deliberately, and include the editions that shrank.

Reliable annotation. Have a second person independently write truth SQL for a sample. Do it for 12 of the 84 items and expect disagreement on around 3 of them, which is 25 per cent. When it happens, the finding is almost never a SQL bug. Two competent people read an ambiguous question differently, both wrote correct queries for the reading they took, and the repair belongs in the wording of the item.

Statistical power. This is where small sets get oversold. Bowman and Dahl put it plainly: "To reliably detect this smaller absolute improvement, though, requires two orders of magnitude more evaluation data." An 84 item set will separate a system scoring 60 per cent from one scoring 80. It will not settle a two point difference, and reporting one as an improvement is the most common misuse of an internal eval set. Turning grades into a rate with an interval around it is Q8's subject and it is the arithmetic that keeps this honest.

Disincentives for biased systems. The original criterion is about social bias, and the event analogue that matters commercially is the silent drop. A query that excludes registrations with a null company field will answer most questions beautifully and undercount exactly the self-service channels. So include items whose true answer depends on the awkward rows, and grade them strictly.

Who can rebuild the set after you leave?

An eval set with no documentation becomes folklore in about six months. Somebody adds items, somebody else changes a truth query, and nobody can say what the set covers.

Gebru and colleagues proposed the fix for datasets generally in Communications of the ACM in 2021, volume 64, pages 86 to 92, with datasheets that ask a standard set of questions across seven areas: motivation, composition, collection process, preprocessing and labelling, uses, distribution and maintenance. The full instrument is heavier than an 84 item eval set needs. The categories are exactly right.

One page, held next to the set. Which channel the questions came from and over what dates. How many were harvested and how many survived clustering. What was excluded, including the personal data you stripped. Who wrote the truth queries and against which schema version. What the comparison conventions are. When the set was last re-derived from a fresh harvest.

Score more than one thing while you are there. Liang and colleagues at Stanford's Center for Research on Foundation Models made the case in 2022 for reporting several measures beside each other so accuracy never travels alone, and for an event assistant the useful companions are the refusal rate and the time to answer. The trade-off between a frozen set and what live traffic is actually asking belongs with offline evaluation against live monitoring in Q13.

Where this stops

A set harvested from your request channel encodes the questions your team has learned to ask. The questions people stopped asking, because the answer used to take three weeks, are absent by construction, and those are often the ones an assistant would be most useful for. Nothing in the harvest surfaces them. Somebody has to sit down and add them deliberately, knowing they are guesses.

The second limit is seasonality. A quarter harvested in March is full of pacing and target-setting. Show week questions are operational, urgent and phrased differently, and a set built in the quiet season under-represents them badly. Re-harvest at least once per show cycle and keep the old set for comparison.

The third is that your truth SQL is unverified code. Until a second person has read each query against the schema, a green run proves the assistant agrees with your queries, and the queries are the part nobody has checked. That is also why the set is worth freezing once it is verified, since running a pinned set across model versions in Q12 depends entirely on the expected answers being right.

Export one quarter of the channel where people ask your data person for numbers, strip everything that is not a question about event data, and count the distinct ones. That count is the size of the set you need, and it will tell you within an hour whether the work in front of your AI layer is a week or a quarter.