Why models get arithmetic wrong and what to compute in SQL instead

A post-show summary says international registrations grew 6.9 per cent. Somebody in the room says they thought it was 14 per cent. A third person, looking at the same two figures, makes it 1.2. All three are defensible readings of the same data, and the model that wrote the summary picked one without saying which. Understanding why models get arithmetic wrong turns out to be two problems stacked on each other: the model cannot reliably divide, and the question it was asked did not specify what to divide by.

The second problem is the bigger one and nobody talks about it.

Three correct answers to one sloppy question

Here are the numbers. In 2025 the show took 9,864 registrations, of which 1,742 were international. In 2026 it took 10,530 registrations, of which 1,988 were international.

Ask for "the change in international registrations" and there are three honest answers.

The count grew from 1,742 to 1,988, which is 246 more, and 246 over 1,742 is 14.1 per cent.

The share went from 1,742 over 9,864, which is 17.66 per cent, to 1,988 over 10,530, which is 18.88 per cent. That is a rise of 1.22 percentage points.

The share grew relatively by 1.22 over 17.66, which is 6.9 per cent.

Three numbers, all correct, describing different things. A model asked the vague question will return one of them, phrased with total confidence, and will not mention that two others exist. Whichever it picks goes in the board pack.

Notice also that the least impressive of the three is the one a reader most likely wants. A 14.1 per cent rise in international registrations sounds like a result. Much of it is the show getting bigger: total registrations rose 666, or 6.8 per cent, and holding the 17.66 per cent share flat against 10,530 registrations would have produced 1,860 international registrations on its own, an increase of 118. That is 48 per cent of the 246 actually gained. The 1.22 point shift in share is the part that reflects a change in who came, and it is the number a marketing director should be judged on.

No prompt engineering fixes this, because it is not a modelling failure. The question was underspecified and somebody has to specify it. That somebody should be whoever owns the metric definition, once, in the query.

Why does a language model get 900 over 6,400 wrong?

Now the second problem, which is the one people expect.

Take three shows in a portfolio. Show A has 1,000 registrations and is 20 per cent international. Show B has 5,000 and is 10 per cent. Show C has 400 and is 50 per cent.

Hand a model those three percentages and ask for the portfolio international share. The available operation, given three percentages and nothing else, is to average them: 20 plus 10 plus 50, over 3, which is 26.67 per cent.

The correct answer needs the counts. Show A contributes 200, show B contributes 500, show C contributes 200. That is 900 international out of 6,400 total, which is 14.06 per cent.

The gap is 12.6 percentage points, and 26.67 is nearly double the truth. A model handed only the percentages will produce the wrong figure while doing nothing wrong, because the information required for the right figure was never in its input. Hand it the counts and it may still err, because the division is now the model's job.

Dziri and colleagues examined that job directly in Faith and Fate at NeurIPS 2023, testing multi-digit multiplication, logic grid puzzles and a dynamic programming problem. Their finding was that transformer models "solve compositional tasks by reducing multi-step compositional reasoning into linearized subgraph matching, without necessarily developing systematic problem-solving skills", with performance decaying rapidly as task complexity rises. Cobbe and colleagues had made a narrower version of the point in 2021 when they released GSM8K, 8,500 grade school word problems, on the grounds that even the largest transformer models failed to reach high accuracy on them.

Multi-digit division against a five-figure denominator is exactly the shape of task that decays.

There is a reason this surprises people, and it is worth naming. A model will describe the correct method fluently. Ask it how to compute a portfolio-wide share and it will tell you, accurately, to sum the numerators, sum the denominators and divide. Then ask it to do that over eight shows and it will produce a number that does not match. The explanation and the computation come out of the same next-token process, and only the explanation is genuinely a language task. Fluency about arithmetic is not evidence of arithmetic, and it is by far the most convincing wrong signal in this whole area.

Showing the working helps and does not solve it

The standard mitigation is to make the model write out its steps. It does improve things, and the reason it improves them is worth being precise about.

Gao and colleagues published PAL at ICML 2023, in volume 202 of Proceedings of Machine Learning Research, pages 10764 to 10799. Their approach uses the model to read the problem and emit a program as the intermediate reasoning, then "offloads the solution step to a runtime such as a Python interpreter". On GSM8K their method beat PaLM-540B using chain-of-thought prompting by 15 points absolute on top-1 accuracy, from a much smaller model.

The gain came from taking the arithmetic away from the model, with the model kept for the part it is good at, which is turning a sentence into a set of operations. That is the design principle to carry into event reporting, and the runtime in your case is the warehouse rather than a Python process.

What to hand the model instead

A short, explicit list of named values, each with its period and its source, and an instruction that it may reference nothing else.

registrations_total, 10530, 2026 edition, fact_registration registrations_total_prior, 9864, 2025 edition, fact_registration international_total, 1988, 2026 edition, fact_registration international_total_prior, 1742, 2025 edition, fact_registration international_share_pct, 18.88, 2026 edition, derived in warehouse international_share_pct_prior, 17.66, 2025 edition, derived in warehouse international_share_change_pp, 1.22, 2025 to 2026, derived in warehouse

Every derived figure is precomputed and named unambiguously. The model's job is now to write English around values it did not calculate, which is a language task and it is good at language tasks.

Two rules make this hold up in practice. Require the output to quote the operands next to any derived figure, so a reader sees "the international share rose 1.22 points, from 17.66 per cent to 18.88 per cent" and can check the subtraction in their head. And name the metric in the variable, since international_share_change_pp cannot be confused with a relative change the way "the international change" can.

The naming discipline is doing more work than it appears to. Once a metric has a variable name, somebody has to decide what that name means, and the deciding is where definitional disagreements surface. Two people will argue for an hour about whether a comp registration counts as international if the delegate's employer is domestic, and that argument is worth having once against a column definition. Left to the model, the argument never happens and the answer changes depending on the phrasing of the question.

What should the model still be allowed to do?

Compare two given numbers and say which is larger. Order a supplied list. Restate a value in words. Describe what a figure means for a decision. Notice that a value is absent from the list and say so.

What it should never do is produce a number that was not in its input. That is a rule you can enforce mechanically: extract every numeral from the output, check each against the supplied values with a small tolerance for rounding, and flag anything unmatched before publication. The check is a dozen lines of code, it runs in milliseconds, and it catches the failure class that survives every other control.

Run it on your last AI-written summary before you build anything else. In my experience the first run always finds something, and it is usually a percentage that reads perfectly.

Where this stops

Computing in SQL moves the arithmetic to a place where it is deterministic. It does not make it right.

A weighted average computed correctly over the wrong denominator is still wrong, and warehouse queries carry definitional errors as readily as models do: comps included in one metric and excluded from another, an edition boundary that splits a two-day show across two rows, a currency conversion applied at the wrong date. The difference is that a query can be read and disputed by a person, and a model's internal division cannot.

The other limit is that precision creates its own trap. A summary reading "18.79 per cent" implies a measurement good to two decimal places, and if your registration file has 300 duplicate records in it, the second decimal is noise dressed as accuracy. Round the output to the precision the underlying data supports and say what that precision is.

Whether the question should have reached a model at all is the prior decision, covered in when a stored query beats generation and in the broader test set out in deciding where AI is the wrong tool. The wider set of controls sits on the AI platform side.

Take the last AI-written summary your team sent and mark every number in it as either supplied or computed by the model. Anything in the second category is the work for this week, and moving it into the query usually takes less time than the argument about whether it matters.