Regression testing AI features when the model underneath keeps changing

An email arrives from your provider giving 60 days notice that the model version your assistant runs on is being retired. Nothing in your codebase changes. No pull request, no diff, no review. On the day of the swap the assistant will answer a question about exhibitor rebooking differently, and the first person to notice will be a show director reading a number they do not recognise. Regression testing AI features exists to move that discovery from the show director to you, and to move it earlier by about two months.

The awkward part is that the usual testing instincts do not transfer cleanly, because the output is prose and the thing you care about is underneath it.

A model upgrade is a code change with no diff

Treat every model version string as a dependency version. That framing is uncontroversial for a database driver and somehow radical for a language model, even though the model has more influence over the output than any library in the stack.

Sculley and colleagues made the general argument at NIPS in 2015, listing the risk factors that make machine learning systems expensive to maintain: "boundary erosion, entanglement, hidden feedback loops, undeclared consumers, data dependencies, configuration issues, changes in the external world, and a variety of system-level anti-patterns". A provider deprecating a model is a change in the external world arriving through a data dependency you do not control, and the undeclared consumers are every report that quietly embeds the assistant's output.

So the first artefact is a pinned model version in configuration, recorded in every run log. The second is a question set that never changes without a decision.

What should a regression suite for an assistant contain?

Sixty questions, pulled from a real log rather than invented, grouped by the work they do. For an event assistant that grouping tends to fall out as counts, cross-edition comparisons, filtered slices, ranked lists, and lookups into free text such as exhibitor feedback.

Ribeiro and colleagues gave the useful vocabulary for this at ACL in 2020 with CheckList, which separates minimum functionality tests, invariance tests and directional expectation tests. All three transfer directly.

A minimum functionality test is the plain version of a question the team asks weekly. How many exhibitors from the 2025 edition have rebooked for 2026. One question, one correct row count.

An invariance test is the same question phrased five ways. How many 2025 exhibitors have rebooked. What is our rebooking count against last year. Rebooked exhibitor count 2025 to 2026. The answer must be identical across all five, and when it is not, you have found something a released feature will hit within a week. In their user studies, practitioners using CheckList wrote about twice as many tests and found close to three times as many bugs as those without it, which is worth knowing before you decide the exercise is overhead.

A directional expectation test changes the question in a way whose effect you can predict. Add "excluding sponsors" and the count must go down or stay equal. It can never go up. That is a check you can assert in code without knowing the right answer, and it catches filter handling faster than anything else in the suite.

Execution accuracy is the score worth keeping

Grade on whether the query returned the correct rows. Li and colleagues built the BIRD benchmark on that principle for NeurIPS 2023: 12,751 question and SQL pairs over 95 databases totalling 33.4 GB across 37 professional domains, scored on execution accuracy. Their reported result is the part to sit with. ChatGPT reached 40.08 per cent execution accuracy on that benchmark against human performance of 92.96 per cent.

Two things follow. Text to SQL over a large, messy schema is hard enough that a model change can move you several points in either direction without anyone noticing from the prose. And the metric that exposes it is the returned rows, because a query can be worded four ways and be correct, or worded convincingly and return the wrong 900 rows.

Store, for every question in the suite, the generated SQL, the row count, and a hash of the sorted result set. The hash is what you diff. Prose is what you read afterwards to understand why the hash moved.

Sixty questions, run twice, diffed

Here is the run that matters, with numbers you can follow.

Run the 60 against the pinned model. Suppose 51 return the correct result set, so execution accuracy is 85.0 per cent. Run the same 60 against the replacement. Suppose 53 are correct, or 88.3 per cent. The headline is an improvement of 2 questions, 3.3 percentage points, and a reasonable person signs off the upgrade.

Now diff at the question level instead of the total. Seven questions that were wrong are now right. Five that were right are now wrong. Net plus two, and 12 changed behaviours, of which 5 are regressions.

That distinction is the entire reason to run the suite. If those 5 include the cross-edition comparison your commercial director runs every Monday, the upgrade has made your Mondays worse while improving your average. Read the 5. Decide whether each one is acceptable. The decision is yours to make and it needs the list in front of you.

Weighting helps here, and it costs nothing. Tag each of the 60 questions with how often it actually gets asked, taken from the log you built the suite from. If your 5 regressions carry weights of 22, 14, 3, 1 and 1 asks a fortnight, they account for 41 asks. If the 7 improvements carry weights of 1, 1, 1, 2, 1, 1 and 2, they account for 9. The unweighted score says the upgrade is better by two questions. The weighted view says it costs you 32 more wrong answers a fortnight than it saves, and the weighted view is the one your users experience.

None of that means block the upgrade. Deprecation windows close whether you approve or not. It means you go into the swap knowing which five questions to fix first, with two months to fix them, instead of finding out on the morning the old version stops responding.

How much of the difference is just noise?

Before you interpret a 2 question improvement, find out how much the suite moves against itself.

Run the 60 questions twice against the same pinned model, same prompt, same temperature. Count how many produce a different result set hash between the two runs. Suppose 4 do. Your self-disagreement rate is 4 out of 60, or 6.7 per cent, which is a swing of about 4 questions from nothing at all.

A 2 question improvement sitting inside a 4 question noise band is not evidence of anything. To resolve a difference that small you need either more questions or more repeats. Running each question three times and taking the majority result cuts the variance materially and triples the cost, which for 60 questions is usually still an afternoon of compute.

Setting temperature to zero helps and does not solve it. Provider-side batching and routing keep a residual, and the honest move is to measure the residual rather than assume it away. The same figure is worth reporting alongside any grading exercise, since a model grader has its own instability on top of this one, which is where the reliability of a model judge comes in.

What to record with every run

• The model version string exactly as the provider reports it, plus the date of the run.
• The prompt template hash, because prompts change more often than models and get blamed less.
• The schema version, since a renamed column will break a question in a way that looks identical to a model regression.
• The result set hash and row count per question, which is the thing you diff.
• The retrieved context, where the answer depends on documents, because a retrieval change and a generation change produce the same symptom and want different fixes.

That last one keeps causing arguments. Separating the two is a measurement problem with its own method, and it belongs with scoring the retriever on its own rather than in the regression suite.

The limit

A pinned suite tests the questions you froze. It says nothing about the ones users started asking last month, and an assistant in daily use drifts toward questions nobody anticipated. The suite ages, and a suite that has been green for six months is usually a suite that stopped covering the product rather than a product that stopped breaking. Rebuilding it from live logs on a schedule is the counterweight, and the trade-off between the two is the subject of offline evaluation against live monitoring.

The second limit is that a green suite tells you the answers did not change. It does not tell you they were right in the first place. If question 14 has been returning the wrong rows since the day you wrote it, the regression suite will faithfully protect that error through every upgrade. Somebody has to verify the expected result sets by hand once, against the warehouse, before the suite has any authority at all. Where a question has a stable, checkable answer, the case for skipping generation entirely and storing the query is strong, which is the argument in when a stored query beats a model, and the broader set of controls sits on the AI platform side.

Start by writing down the model version string your assistant is running against today and where it is configured. If that takes more than five minutes to find, the pinning work comes before the eval set, because a suite you cannot tie to a version tells you nothing when it goes red.