When rules beat a model for questions an event team asks daily

The question-and-answer box goes in, everyone is pleased, and three weeks later the show director asks the same thing she asked yesterday and gets a different number. Working out when rules beat a model is worth doing before that happens, because for the questions an event team asks every day the answer is usually that a stored query wins on every dimension anyone cares about: correctness, speed, cost and the ability to explain the result to a board.

This is not an argument against generation. It is an argument about which questions belong on which path.

What the enterprise benchmarks actually report

Text to SQL demos are run against tidy schemas. Your warehouse is not one.

Lei and colleagues built Spider 2.0 for ICLR 2025 out of 632 real enterprise text-to-SQL workflow problems, drawn from databases that "often contain over 1,000 columns" and live in systems like BigQuery and Snowflake. Their code agent framework built on o1-preview solved 21.3 per cent of them. The same paper reports 91.2 per cent on the older Spider 1.0 benchmark and 73.0 per cent on BIRD.

That gap between 91.2 and 21.3 is the difference between a benchmark schema and a real one, and it is the single most useful number in this whole area. Li and colleagues had already found something similar with BIRD at NeurIPS 2023, where ChatGPT reached 40.08 per cent execution accuracy against human performance of 92.96 per cent on the same tasks.

Roughly one in five, on the benchmark closest to what an event data warehouse looks like. A saved query clears that bar trivially, and it clears it every time rather than four times in five.

Counting the repeats in a fortnight

Do the counting before you decide anything. It takes an hour and it settles most of the argument.

Log every question the team asks for two weeks, whether it goes through a tool or arrives as a message to the data person. Suppose you collect 214. Now normalise each one: strip the show name, the date range and the segment, and reduce it to its shape. "How many exhibitors from Autumn 2025 have rebooked" and "rebooking count for the spring show" are the same shape with different parameters.

Suppose the 214 collapse to 38 distinct shapes. Sort by frequency. In every log I have seen the distribution is steep, so assume the top 12 shapes account for 141 of the 214 asks. That is 65.9 per cent of everything asked, covered by 12 parameterised queries.

Write those 12. At 40 minutes each, including checking the result against the warehouse by hand, that is 8 hours. Write all 38 and you are at about 25 hours, and you have covered 100 per cent of a fortnight's questions with SQL somebody has read and signed off.

Twenty-five hours is three days of one person's time to make the most-asked questions in the business exact and permanent.

What does a stored query buy that generation cannot?

Four things, and only one of them is accuracy.

It gives the same answer twice. A show director who asks on Monday and again on Thursday gets numbers that differ only because the data changed, which means she can trust the difference. Generated SQL varies across runs, and the variation is invisible in the prose.

It is auditable. Somebody read the query, checked the joins, and confirmed that "rebooked" means what the commercial team means by it. That definition then lives in one place. When finance disputes the number, the conversation is about the query rather than about whether the model understood.

It costs nothing to run. Two hundred and fourteen questions a fortnight at, say, 12,000 prompt tokens each for schema and context is about 2.57 million tokens, repeated every fortnight forever. The stored query costs the compute of the query itself.

It answers in the time the database takes. A generated answer waits for a model to write SQL, waits for the query, then waits for prose. During show week, when the question is being asked from a phone in a hall, the difference between 400 milliseconds and 12 seconds decides whether the tool gets used at all.

What a stored query has to carry

A stored query is more than SQL in a file, and the extra parts are what make the router possible.

• The question shapes it answers, written out in the phrasings people actually use, including the clumsy ones. This is the matching surface and it should be maintained by whoever hears the questions.
• The parameters and their types, typically show, edition, date range and segment, with a default for each so a question that omits one still runs.
• A prose definition of every term in the answer. What counts as a rebooking, whether comps are in the registration total, which badge types are excluded. This is the part that gets argued about at year end.
• A field synonym list, mapping the words people say to the columns that hold them. Net square feet, NSF and net square metres point at the same measure with a unit conversion between two of them. Rebook, renew and resign point at one column.
• A last-verified date and an owner, because a definition without a name attached is a definition nobody will update.

The synonym list is the piece most teams leave out and the piece that does the most work. Forty entries, maintained by the person who writes the post-show report, turns a brittle matcher into one that handles how the commercial team actually talks. It is also reusable: the same mapping feeds the router, the keyword index over your documents, and any later attempt at a semantic model behind the question box.

Routing between a stored query and a generated one

The design that works is a router, and the router is more valuable than either path.

An incoming question gets matched against the stored set. Matching can start crude: keyword overlap plus a small embedding similarity, with a threshold. Above the threshold, extract the parameters, run the stored query, and answer. Below it, fall through to generation with a visible label saying the answer was generated rather than retrieved from a checked query.

Two properties make this worth building. Every fall-through is logged, so the stored set grows from evidence instead of from guesses about what people will ask. And the label is honest, which means a reader knows which answers carry a signed-off definition and which do not.

Set the matching threshold conservatively. A stored query fired on a question it does not actually answer is worse than a generated one, because the answer arrives with the authority of something that was checked. Where the router is unsure, it should ask which of two stored questions the person meant, and that clarification is cheap.

When should you let the model write the SQL?

When the question is genuinely new and the cost of a wrong answer is low.

An analyst exploring a hypothesis on a Tuesday afternoon benefits enormously from generated SQL, because she can read the query, spot that it joined on the wrong key, and fix it. The model has saved her twenty minutes of typing and she is the check.

The same query embedded in a report to 600 exhibitors has no check in it at all. That is the same tool in two situations with completely different failure costs, and the sensible policy distinguishes them by who reads the output rather than by how good the model is.

The other honest case for generation is the long tail. Thirty-eight shapes covered a fortnight; the thirty-ninth arrives eventually and nobody is going to write a query for a question asked once. Generation handles that, with the label attached.

The limit

Stored queries rot, and they rot silently.

A column gets renamed in a warehouse migration and the query errors, which is the good case because somebody notices. The bad case is a definition change: the commercial team decides that a rebooking now counts from the date of signature rather than the date of invoice, updates the dashboard, and the stored query keeps returning the old definition to anyone who asks the assistant. Two numbers, both defended, both from the same warehouse.

The defence is to treat the stored set as code, with the same review, the same tests and the same ownership as anything else in the warehouse. Each query needs a named owner, a definition in prose next to it, and a test that runs on a schedule. Keeping a question set stable and re-running it after every change is the same discipline described in regression testing when the model changes underneath, and it applies to the stored path as much as the generated one.

The second limit is that a stored query answers the question it was written for. It has no ability to notice that the question is the wrong one, which a good analyst does constantly. Automating the top 12 shapes frees that person to ask better questions and does not replace the asking. Where the arithmetic inside the answer comes from matters just as much, and that is the subject of why a model gets a percentage wrong. The broader test for whether a problem suits a model at all sits in deciding where AI is the wrong tool, and the wider set of choices lives on the AI platform side.

Start the log this week. One shared document, every question anybody asks the data team, for fourteen days, with no attempt to tidy them. Count the distinct shapes at the end. If the top ten cover more than half, you know exactly what to build and roughly how long it takes.