What text to SQL accuracy benchmarks tell an event data team

The slide says 90 per cent accuracy. Nobody in the room asks 90 per cent of what, measured how, on whose database, and the meeting moves on to pricing.

Text to SQL accuracy benchmarks are public, the methods are documented, and reading them properly takes about twenty minutes. It is twenty minutes worth spending before anyone promises a show director that they can ask the warehouse questions in English.

What the benchmark number is actually measuring

BIRD is the one to start with. It was published at NeurIPS 2023 as a spotlight paper by researchers from Alibaba's DAMO Academy with the University of Hong Kong and the University of Illinois Urbana-Champaign, and it holds 12,751 question and SQL pairs across 95 large databases totalling 33.4 GB, drawn from more than 37 professional domains.

The headline metric is execution accuracy. Both the generated query and a hand-written reference query are run against the same database, and the answer counts as correct when the two result sets match. That choice matters. A generated query can look nothing like the reference, use a different join order, a different subquery structure, a different set of aliases, and still be right, and execution accuracy gives it the credit. It also means a query that throws an error scores zero rather than partial marks, which is the correct treatment.

Two things follow from the metric that people miss. A query can be marked correct while being correct only on that database, because the two result sets happened to agree on the rows present. And a query marked wrong might be wrong by one row, or wrong by returning the entire table.

The first of those is the one that bites event data. An inner join between a registration table and a registration status table returns the same rows as a left join for as long as every registration has a status. The morning a bulk import lands 400 rows with a null status, the two queries diverge and only one of them was ever right. Execution accuracy scored on the data that existed at benchmark time cannot see a defect that only appears when the data changes shape, and registration data changes shape constantly: new registration types, a channel that stops populating a field, an exhibitor allotment upload that arrives with a different structure from last year's.

Why is the human baseline more useful than the leaderboard?

BIRD reports a human execution accuracy baseline of 92.96 per cent, produced by data engineers and database students working the same questions. That figure is the one I would put on the slide instead of the vendor's.

Skilled people, given the schema and the question, get about 7 per cent of these wrong. Not through carelessness, but because a question phrased in English against an unfamiliar 33 GB database is genuinely ambiguous, and two competent engineers make different defensible choices. If your internal target for a generated query is 100 per cent, you have set a target that humans do not hit on the same task.

The best system listed on the BIRD leaderboard sits at 81.95 per cent, recorded in September 2025 for AskData with GPT-4o, submitted by AT&T's CDO DSAIR team. The gap to the human baseline is 11.01 percentage points. That number, the distance between the best published system and skilled humans on the same questions, is the most honest single figure in this whole area.

The gap, mapped onto one show week

Now put it on your own calendar, because a percentage on a leaderboard means nothing until it becomes a count of conversations.

Say your audience acquisition and sales teams together ask 40 questions a week of the data function during a campaign, and the campaign runs twelve weeks. That is 480 questions. At 81.95 per cent execution accuracy, 393 come back right and 87 come back wrong. Round it to 86 and the point is unchanged.

Eighty-six wrong answers in a quarter sounds catastrophic until you notice how few of them matter, and it sounds acceptable until you notice which ones do. Suppose one question in five leads to a decision: a spend reallocation, a floor plan change, a message to exhibitors. That is 96 decision-driving questions, of which roughly 17 carry a wrong number. Seventeen bad decisions in a quarter is a different conversation from 86 bad answers, and it is the conversation to have.

Those 480 questions are also not spread evenly. In most shows the volume concentrates in the four weeks before doors and the two weeks after close, so perhaps 300 of the 480 arrive in six weeks, which puts around 54 wrong answers into the period when nobody has time to check anything. Benchmark accuracy is a flat rate across a year. Your exposure to it is seasonal, and it peaks exactly when the review capacity is lowest.

The arithmetic gets worse when one question needs several queries. A question like "which exhibitor categories improved rebooking against last edition, and what did those exhibitors spend on space" is three retrievals joined by a comparison. At 82 per cent per query, all three land correctly 0.82 times 0.82 times 0.82 of the time, which is 55.1 per cent. A little over half. Multi-step questions are exactly the ones people most want to ask a box, and they are where the published single-query accuracy stops describing what you will experience.

Which benchmark resembles your warehouse?

This is where most vendor numbers quietly fall apart, and it is worth being specific about why.

The original Spider benchmark, released by Yale in 2018, has 10,181 questions across 200 databases covering 138 domains. Those databases are small and clean. XLANG Lab's Spider 2.0, presented at ICLR 2025, was built precisely because that was no longer a useful proxy: it holds 632 real-world text-to-SQL workflow problems where individual databases carry over 1,000 columns and answering often requires reading documentation and metadata first. GPT-4o scores 86.6 per cent on the original Spider and 10.1 per cent on Spider 2.0.

A portfolio registration warehouse is closer to the second. Count your own columns before deciding which figure applies to you. A core registration schema with an identity layer, an exhibitor contract layer and five years of edition history is comfortably into the hundreds of columns, carries several plausible date fields per table, and encodes definitions that exist nowhere in the column names. If a vendor quotes a Spider 1.0 style number at you, they are quoting the tutorial database. The semantic model work that narrows your schema is Q1's subject and it is the main lever you have for moving your own number.

How to ask for a figure you can use

Four questions get you further than any demo.

Which benchmark, and which split? A number with no benchmark behind it is a marketing figure. A number from a development split that the system was tuned on is worse than no number.

What is the denominator? Execution accuracy over all questions is one thing. Accuracy over the questions the system chose to attempt, with refusals excluded, is a completely different thing and reads about fifteen points higher.

Was the schema in training data? Public benchmark databases have been on the internet for years. Performance on a schema the model has memorised tells you nothing about performance on yours.

What does it score on our schema? The only figure that binds. Give them 60 of your own questions with hand-written reference SQL and let them run it. If they will not, that is the answer.

The refusal question deserves a second look, because a system that declines to answer is doing something valuable and the leaderboards have no column for it. Two systems both reporting 82 per cent are very different products if one attempts every question and the other attempts 70 per cent of them and refuses the rest. The second gives you 82 per cent of 70 per cent, which is 57 answers per hundred questions, with 30 honest refusals and 13 wrong answers. The first gives you 82 right and 18 wrong. Which you prefer depends on whether a wrong answer costs more than no answer, and in reporting that goes to exhibitors it usually does.

Where this stops

Benchmarks measure systems on other people's data, and every one of the numbers above is a bound rather than a prediction. Your accuracy will be different, and the direction is not fixed: a tightly curated ten-table subject area with good synonyms can beat the published figures, and a raw warehouse will do far worse than them.

There is a second limit that matters more. Execution accuracy compares result sets against a reference query written by somebody who had already decided what the question meant. It says nothing about whether that interpretation was the one the asker intended. A benchmark cannot measure the failure where the query is technically perfect and answers a different question, which is the failure event teams actually hit, because words like attendee and rebooking carry a definition the schema does not hold. Catching that needs the query and its filters shown beside the answer, which Q3 covers.

Contamination is the third. Nobody can currently rule out that a published benchmark's questions sit somewhere in a model's training data, and the leaderboards themselves are a moving target as systems are tuned against them.

Write 20 questions your team actually asked last month, write the reference SQL for each one by hand, and run whatever you are evaluating against them. Twenty is enough to tell a 50 per cent system from an 80 per cent one, it takes a day, and it is the only accuracy figure about your own AI reporting layer that anyone should quote in a board pack. Growing that set into something you can rerun every time the model changes is a separate exercise, and Q10 sets it out.