Choosing a model for summarisation of event aggregates without guessing
Choose a summarisation model by running a fixed set of your own event aggregates through each candidate and scoring grounding, cost per answer and seconds per answer in one table. Published hallucination rankings are a shortlist tool. They rarely match general capability leaderboards and they never match your prompt.
The usual way a model gets picked for an event reporting pipeline is that somebody read a benchmark table, or somebody already had an account with a vendor, or somebody tried two prompts on a Friday and one of them read better. Then 900 exhibitor summaries go out on that basis.
Choosing a model for summarisation deserves more than that, and it takes about two days. The reason it takes two days rather than two hours is that the property you care about, whether the prose stays faithful to the aggregates you handed over, is not the property the general leaderboards measure, and the ordering on the one you care about is genuinely different.
Where the published rankings help and where they stop
Vectara has been publishing a hallucination leaderboard since 6 November 2023, when it released the Hughes Hallucination Evaluation Model as open source. The method is narrow and useful: feed each model a set of source documents, ask for a summary using only facts present in the document, and score whether the summary is consistent with its source. The current edition, updated in May 2026, runs over 7,700 curated articles at temperature zero and reports a hallucination rate, a factual consistency rate, an answer rate and an average summary length per model.
Two things in that table are worth an event data lead's attention.
The first is that model size and general reputation predict the ordering badly. In the May 2026 edition, a nano-class model from one major vendor sits at 3.1 per cent while the mini-class model from the same vendor and the same release date sits at 5.5 per cent. Compact open-weight models cluster near the top: one 12 billion parameter model records 4.4 per cent and an 8 billion parameter model records 4.8 per cent, ahead of several models many times their size. If your shortlist came from a general capability ranking, it is a shortlist for a different question.
The second is the answer rate column, which is the trap. One model in that table posts a 4.3 per cent hallucination rate with an answer rate of 62.7 per cent, and another posts 3.7 per cent with an answer rate of 80.7 per cent. A model that declines to produce a summary cannot produce an unfaithful one. Read hallucination rate on its own and you will shortlist the model that says least, then discover during the show that a fifth of your exhibitors got no paragraph.
Use the leaderboard to build a shortlist of three. Do not use it to pick one, because it is measuring news and technical articles rather than a table of scan counts with your instructions wrapped around it.
Grade your own aggregates, three candidates, sixty each
The eval set is the work, and it is reusable forever, which is what makes it worth building properly once.
Take 60 real inputs from your own pipeline. Each one is the exact aggregate bundle a summary would be generated from: named values, periods, source tables. Cover the ordinary cases and deliberately include the awkward ones, because the awkward ones are where candidates separate. A show with a date change so the prior year comparison is not like for like. An exhibitor with a missing value. A figure that fell. A first-time exhibitor with no prior edition at all. A tiny stand where every count is under ten and percentages are silly.
Run all 60 through each of your three candidates with the same prompt. That is 180 outputs.
Then grade. Four scores per output, all binary or near-binary, because scales invite disagreement between graders:
- Every figure traceable. Does each number in the prose appear in the input bundle, or follow from it by arithmetic that is shown?
- No invented claim. Does the prose assert anything the bundle does not support, including directional claims about causes?
- Missing data handled. Where an input is absent, does the output say so or skip it, instead of filling the gap?
- Arithmetic correct. Where the output computes, does the computation check out?
Grade the 180 by hand. It is roughly four hours of somebody's time and it is the only part of this that cannot be automated on the first pass. If you later want a model to do the grading, you now have 180 human labels to check it against, which is the right order to do it in and is Q11's subject.
What goes in the comparison table
One table, three rows, and every column populated from the same run.
| Candidate | Traceable | No invented claim | Missing handled | Arithmetic | Cents per answer | Seconds to complete |
|---|---|---|---|---|---|---|
| A | 58/60 | 60/60 | 41/60 | 57/60 | 3.9 | 4.1 |
| B | 60/60 | 59/60 | 55/60 | 60/60 | 11.2 | 6.8 |
| C | 54/60 | 56/60 | 38/60 | 49/60 | 1.1 | 2.3 |
Those figures are illustrative, produced to show the shape of the decision rather than measured from any real run. The shape is the point.
Keeping cost and speed in the same table as the quality columns is deliberate, and it follows the design of the broader evaluation work. When Stanford's Center for Research on Foundation Models published Holistic Evaluation of Language Models in 2022, efficiency sat inside the scored metric set: the paper reports seven metrics per scenario, "accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency". Grading speed and spend at the same time as quality is what stops a team picking on quality and discovering the operating cost afterwards.
Candidate C is a fifth of the price and twice as fast, and it fails arithmetic 11 times in 60, which is 18 per cent of summaries carrying a wrong computed figure. That is disqualifying for anything an exhibitor reads, and it is entirely fine for an internal draft that a person will rewrite. Candidate A is close to B on the traceability scores and fails missing-data handling 19 times, which is the single most common real defect in event summaries because exhibitor data is patchy by nature. B costs 2.9 times A per answer.
Work the cost through before deciding. At 900 exhibitors, A costs 35 dollars a run and B costs 101. The difference is 66 dollars. Reviewing the 19 extra defective outputs from A at three minutes each of a reviewer's time is 57 minutes, which at 45 dollars an hour is about 43 dollars, plus whatever it costs when one slips through. On those numbers B and A are close enough that the tie goes to the one with fewer defects, and anyone arguing the 66 dollars is significant should be shown the full cost picture in Q24, where the model spend is a rounding error next to review time.
Should the missing-data column decide it?
For event aggregates, more often than any other column, yes.
Exhibitor data is incomplete in ways that are structural. First-time exhibitors have no prior edition. Shared stands split scans unpredictably. A hall with a broken scanner produces a real zero and a missing value that look identical in a bundle unless you distinguish them. Across a portfolio, the share of exhibitors with at least one absent input is routinely a quarter or more.
A model that quietly interpolates when a value is absent will produce its most confident and most wrong sentences exactly on those exhibitors, and those are disproportionately the small and new accounts your sales team is trying to retain. A model that says a comparison is unavailable produces a less satisfying paragraph and never invents a growth rate for a company that did not exhibit last year.
Test it directly. Of your 60 inputs, make sure at least 15 have a deliberate gap, and score that subset separately. The aggregate number across all 60 will hide a candidate that handles the easy 45 perfectly.
How often should this be redone?
On a model version change, on a prompt change large enough to alter the output shape, and otherwise every six months.
Keep the 60 inputs frozen and version-controlled so runs are comparable across time. Add to the set when you find a new failure in production, which is the natural way an eval set grows and the reason it gets better than anything you could design up front.
One warning about rerunning. The graders drift too. A person who has read 500 of these summaries stops seeing the missing-data failures, which is the same effect that makes approval rates unreliable in Q23. Rotate graders between runs, or seed a handful of known-bad outputs into the grading set and check they get marked.
Where this stops
Sixty examples is a small sample and the intervals around these rates are wide. A 58 out of 60 against a 60 out of 60 is not a real difference. A 38 out of 60 against a 55 out of 60 is. Use the eval to eliminate candidates and to catch disqualifying behaviour, and stop short of ranking two close finishers, because the ranking will not replicate.
The deeper limit is that this measures faithfulness to the bundle you handed over and nothing else. A candidate scoring 60 out of 60 on every column will write a perfectly grounded paragraph from an aggregate that was computed wrong, and it will write it more persuasively than a weaker model would. No amount of model selection reaches that, and the effort belongs upstream in the measurement layer.
There is also a scope limit worth naming. This procedure compares candidates for narrative summarisation of aggregates. Classification, extraction and tagging are different tasks with different winners, and running one bake-off for all of them will land you on an expensive model doing work a small one does equally well, which is the argument in routing by task at Q28.
The step this week is to collect the 60 inputs. Pull the exact aggregate bundles your current pipeline sent to a model for the last show, pick 60 covering the awkward cases, and save them as files in a repository. That is the durable asset. The grading can wait until you have a reason to compare something, and by then the hard part is done. The pillar covers the surrounding decisions on running AI over event data.
Questions people ask about choosing a model for summarisation
- Do general capability leaderboards predict summarisation grounding?
- Poorly. Vectara's hallucination leaderboard, which has measured summarisation faithfulness with its own evaluation model since November 2023, regularly places compact models above larger ones from the same family. In the May 2026 edition a nano-class model records 3.1 per cent while a mini-class model from the same vendor records 5.5 per cent.
- How many graded examples do you need to compare models?
- Sixty per candidate is a workable floor for a first pass. At that size a difference of one or two percentage points is noise and a difference of ten is real, which is usually enough to eliminate a candidate. Push to a few hundred only for the final two, and spend the effort on covering edge cases instead.
- What is the answer rate trap in a hallucination leaderboard?
- A model can post a low hallucination rate by declining to summarise. Vectara publishes an answer rate alongside the hallucination rate for exactly this reason, and the May 2026 table shows one model at 4.3 per cent hallucination with an answer rate of 62.7 per cent. Read the two columns together or you will pick a model that stays quiet.
Related reading
- Automation bias in AI review and how to keep approvals honest
- Working out the real cost of AI features on an event platform
- Using small models for narrow tasks inside an event data pipeline