Cutting the latency of AI generated answers during show week

On the Tuesday of a show, an operations lead standing at the registration desk asks how today's scans compare with the same hour yesterday. She has a radio in her other hand and someone waiting for her. Forty seconds later the answer arrives on a screen she has already stopped looking at.

That is a latency failure rather than an accuracy failure, and it will kill an internal tool faster than a wrong number will. The latency of AI generated answers is the property people judge the system on in the two weeks a year when they most need it, and it is usually the last thing anyone measures.

Two numbers, never one

Report time to first token and time to complete separately. An average response time is the mean of two distributions with different causes, and improving it tells you nothing about which one moved.

Time to first token is everything that happens before the model emits a character: routing, authentication, retrieval, prompt assembly, queueing at the provider, and the prefill pass over the input. Its enemies are large inputs and slow retrieval.

Time to complete is time to first token plus generation. Generation time is close to linear in output length, so a 380 token answer at a sustained 45 tokens per second takes about 8.4 seconds of generation whatever else you do.

Those numbers respond to different interventions. Trimming the prompt by 4,000 tokens moves the first and barely touches the second. Cutting the answer from 380 tokens to 160 moves the second and does nothing for the first.

What counts as slow on a show floor?

Jakob Nielsen set out the three thresholds in Usability Engineering in 1993, and they have held up well enough that they are still the ones to design against. One tenth of a second is "the limit for having the user feel that the system is reacting instantaneously". One second is "the limit for the user's flow of thought to stay uninterrupted". Ten seconds is "the limit for keeping the user's attention focused on the dialogue", beyond which people go and do something else.

Nothing involving a language model will hit a tenth of a second. The two targets that matter are the other two, and splitting the measurement is what lets you hit them separately: first token inside a second, complete inside ten.

On the show floor those limits bite harder than they do at a desk. Someone standing up, holding a radio, with a queue forming, is at the top of Nielsen's multiple-task case, and ten seconds is generous. I would design the on-site version for first token inside 800 milliseconds and complete inside six, and accept a shorter answer to get it.

A worked example: taking an answer from 11 seconds to 3

Instrument one question end to end and the budget falls out. Take "how do today's scans compare with the same hour yesterday", answered from the warehouse.

Retrieval, running two aggregate queries against fact tables at show scale, 900 milliseconds. Prompt assembly and serialisation, 120 milliseconds. Provider queue and prefill on an 11,000 token input, 1,400 milliseconds. That is a time to first token of 2,420 milliseconds. Generation of 380 output tokens at 45 per second, 8,440 milliseconds. Time to complete, 10,860 milliseconds.

Both numbers are over budget. Three changes, in order of how much they return.

Precompute the aggregates. The comparison the operations lead wants is hour-by-hour scans for today and yesterday, which is a fixed shape known in advance. A job that materialises hourly scan counts every five minutes turns a 900 millisecond aggregate scan into a 40 millisecond lookup on a small table. That is 860 milliseconds off the first number for one scheduled job.

Cap retrieval depth. The 11,000 token input is mostly context that was retrieved because the retriever was configured to fetch twenty passages and nobody revisited it. Cap at six and the input drops to roughly 4,000 tokens, taking prefill from 1,400 to about 600 milliseconds. Time to first token is now 40 plus 120 plus 600, which is 760 milliseconds, inside Nielsen's one second.

Shorten the answer. A comparison of two hourly figures does not need 380 tokens. Instruct for at most 90, which is 2 seconds of generation. Time to complete lands near 2,760 milliseconds.

Eleven seconds to under three, and the largest single contributor was the answer being four times longer than the question deserved.

Which of these can you actually change during show week?

Prompt length and answer length, immediately. Precomputation, if the job already exists. Model choice, if you built the switch before the show. Retrieval depth, in a config change.

What you cannot change during show week is the shape of your warehouse, so the precomputation has to be built before doors open. The list worth materialising is short and predictable: registrations and scans by hour and by day, current totals against the same point in the prior edition, entries by hall, and a session or theatre attendance count if you have one. Those five cover most of what anyone asks between Monday and Thursday.

There is a related saving in never regenerating an answer whose inputs have not moved, which is the caching decision that Q26 takes on properly. During show week the inputs move constantly, so the cache window is minutes rather than days, and the two techniques stack: precompute the aggregate, cache the prose against the aggregate's hash, and a repeated question inside the same five minute window returns without a model call at all.

What streaming actually buys you

Streaming tokens as they generate is the cheapest usability change available, and it is worth being precise about what it buys.

Total time is the same or a shade worse. What changes is that the reader begins at the first token, so a 2.8 second answer feels like waiting 0.8 seconds and then reading. Against Nielsen's thresholds, streaming is how a system that cannot beat one second to completion still keeps a user's flow of thought.

It carries one real hazard. A streamed opening sentence is visible before the model has produced the sentence that qualifies it, and a reader who has already turned away has taken the unqualified version with them. If an answer may end in "this figure excludes hall 3, where the scanner was down until 10:40", that caveat needs to arrive at the top of the answer or in a fixed field outside the streamed prose. Ordering the content so the load-bearing parts come first is a prompt design question, and where to put the facts you want quoted is Q30's subject.

Efficiency belongs inside the score

Speed tends to get treated as something to tune once the answers are good. The evaluation literature has been treating it as part of the score for years.

When Stanford's Center for Research on Foundation Models published Holistic Evaluation of Language Models in 2022, it measured seven metrics per scenario, listed in the paper as "accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency". Efficiency graded alongside accuracy, on the same runs, is the right pattern to copy.

For an event team that means your evaluation suite records seconds to first token and seconds to complete on every graded example, so when you compare candidates you are looking at quality and speed in the same table. A model two points better on grounding and three seconds slower to first token is a genuine trade during show week and an obvious win in the quiet season. You cannot make that call from a leaderboard, and the cheap-and-fast option often wins the narrow tasks, which is what small models are for in Q28.

Where this stops

Every technique here trades something. Precomputation trades freshness, because a five minute materialisation is five minutes stale, and on a show floor five minutes can span the difference between a queue forming and a queue clearing. Put the timestamp of the underlying aggregate in the answer and let the reader judge.

Capping retrieval depth trades recall. Six passages instead of twenty is faster and will occasionally miss the passage that held the answer, and the failure is silent, because a confident answer built from five relevant passages and one irrelevant one looks exactly like a good answer.

Shortening the output trades nuance, and this is the one I would watch most carefully. Compression is where caveats die. A 90 token answer that omits the scanner outage is faster and worse than a 200 token answer that mentions it.

The honest position is that latency work has a floor set by your warehouse. If a query against your scan facts takes four seconds because nobody has indexed them for that access pattern, no amount of prompt trimming reaches Nielsen's one second, and the fix is an index rather than a model. That work is also cheap and dull and reusable by everything else in the building, which is a reasonable argument for doing it first.

Start by adding two fields to whatever you log per AI answer: milliseconds to first token, and milliseconds to complete. Run a show week with those fields populated and look at the ninety-fifth percentile of each rather than the mean. Whichever of the two is furthest past its Nielsen threshold is the one to work on, and the answer is often not the one the team assumed. The surrounding decisions about how these features get built sit with the pillar, and the money side of the same trade is the cost per answer in Q24.