Working out the real cost of AI features on an event platform

The question that produces the wrong answer is "what does the AI cost us a month". The invoice is a number, it goes on a slide, and it tells nobody whether the feature is worth having. The cost of AI features becomes decision-grade only when it is priced per answer, because per answer is the unit that scales when the portfolio adds a tenth show or the exhibitor count goes from 600 to 900.

Per month also hides the two lines that usually dominate. Retries and evaluation calls inflate the token spend by a factor nobody put in the estimate, and the human minutes spent reviewing the output cost more than the tokens by an order of magnitude that surprises people the first time they multiply it out.

The five lines that make up one answer

Price a single unit of output. For an exhibitor performance summary, one answer is one exhibitor's paragraph.

• Input tokens. Everything the model reads: system prompt, instructions, the aggregates, any few-shot example. This is the line people underestimate, because the aggregates look small and the instructions are long.
• Output tokens. What it writes, priced several times higher than input at most vendors.
• Retries. Calls that produced nothing usable. Timeouts, rate limit backoffs, schema validation failures where the output did not parse, and the second attempt after a grounding check rejected the first.
• Evaluation calls. If a second model grades the output for grounding before it reaches a person, that is another call with its own input and output, and its input includes both the source aggregates and the draft.
• Human minutes. The reviewer. Priced at a loaded hourly cost, which is salary plus employment costs plus the overhead your finance team already uses for internal rates.

Four of those five are metered by somebody who sends you an invoice. The fifth is metered by nobody, which is exactly why it goes missing.

A worked example: 900 exhibitor summaries

Take a portfolio show with 900 exhibitors, one summary each, generated after the show closes.

Each summary sends 8,000 input tokens and produces 600 output tokens. The input is large because it carries the instructions, the definitions of every metric, a worked example of the house style, and the exhibitor's aggregates with their source labels.

Input: 8,000 times 900 is 7.2 million tokens. Output: 600 times 900 is 540,000 tokens.

Assume list prices of 3.00 US dollars per million input tokens and 15.00 per million output. Those are assumptions for the arithmetic; substitute your own. Input costs 7.2 times 3.00, which is 21.60. Output costs 0.54 times 15.00, which is 8.10. The generation pass is 29.70 dollars, or 3.3 cents per summary.

Now the lines that get left out.

Retries at 12 per cent of calls adds 108 extra generations, which is 12 per cent of 29.70, so 3.56 dollars.

A grounding check by a second model reads the aggregates plus the draft, call it 2,500 input tokens, and writes a short verdict, call it 150 output. Across 1,008 calls including the retried ones that is 2.52 million input and 151,200 output. At a cheaper model rate of 0.50 and 2.00 per million, that is 1.26 plus 0.30, so 1.56 dollars.

Total model spend: 34.82 dollars for 900 summaries, or 3.9 cents each.

Then the reviewer. Ninety seconds per summary is 81,000 seconds across the batch, which is 22.5 hours. At a loaded cost of 45 dollars an hour that is 1,012.50 dollars, or 1.13 dollars per summary.

The human line is 29 times the model line. Everything you might do to shave tokens off the prompt is competing for a slice of 3.9 cents, while a design change that cuts review from 90 seconds to 25 saves 730 dollars on one show. That ratio is the single most useful output of doing this arithmetic, and it is why the approval screen design in Q21 is a cost decision as much as a governance one.

Which of these numbers should you actually measure?

Three of them, and none of the three is hard to get.

The ratio of model calls to delivered answers, over a real week. Divide total completions logged by the number of outputs that reached a person. A feature everyone describes as one call per answer commonly runs at 1.2 to 1.4 once timeouts and validation failures are counted, and if yours is at 2.0 there is a bug rather than a budget problem.

Median input tokens per call, from your own logs rather than from the prompt template. Templates grow. A prompt that started at 3,000 tokens acquires an extra instruction every time somebody fixes an output problem, and nobody deletes the instruction that the fix replaced.

Median seconds a reviewer spends per item, taken from the gap between consecutive approvals within a batch. This is the number with the largest financial consequence and the one almost nobody logs.

What does the runaway case look like?

OWASP's 2025 edition of the Top 10 for LLM Applications lists Unbounded Consumption as LLM10:2025, and it is the entry to read before shipping anything that loops over a portfolio. Among the attack vectors it names are Denial of Wallet, which exploits pay-per-use pricing to inflict financial damage, along with continuous input overflow, resource-intensive queries and variable-length input flood. Its recommended controls include rate limiting with user quotas, operational timeouts, dynamic resource monitoring and graceful degradation under load.

Most event platforms will never meet a deliberate attacker on this. They will meet the accidental version, which produces the same invoice.

The accidental version looks like a retry loop with no ceiling around a step that started failing at nine in the evening, or a join that fans out so a run that should process 900 exhibitors processes 900 times the number of their booth staff. Both are boring bugs. Both are unbounded consumption in the sense that matters, which is that nothing in the system had a number above which it stopped.

The control worth building first is a per-run cost ceiling that aborts and alerts, expressed in the currency of your invoice rather than in tokens. Nobody can reason about 40 million tokens. Everybody can reason about a run that would have cost 260 dollars when the last four cost 35.

Set the ceiling from your own history rather than from a policy discussion. Take the median cost of the last twenty runs of that job, multiply by three, and use that. On the 900 exhibitor example the median run is around 35 dollars, so the ceiling lands near 105, which is loose enough that a show with more exhibitors than usual passes and tight enough that a fanned-out join stops in the first minute. Review the ceiling when the portfolio grows, because a job that legitimately triples in size will otherwise start failing for the right reason at the wrong time.

Two smaller controls belong next to it and cost almost nothing. A retry cap per item, so a single failing record cannot loop, and a wall-clock timeout on the whole run. Both are boring, both prevent the specific incidents that produce a surprise invoice, and neither needs anyone to be watching at nine in the evening.

Efficiency belongs inside the evaluation

Cost tends to get treated as an operational concern that arrives once the feature works. The evaluation research does not treat it that way.

When Stanford's Center for Research on Foundation Models published Holistic Evaluation of Language Models in 2022, efficiency sat inside the scored metric set alongside quality measures. Their abstract puts it plainly: they measure seven metrics, "accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency", across each core scenario. Efficiency is one of the seven, graded at the same time as the rest.

For an event platform that argues for a single table when you compare candidates, with cost per answer and seconds per answer sitting next to grounding quality, so the trade is visible while the decision is being made. Seconds per answer has its own consequences during show week, which Q25 takes up as the latency problem rather than a cost one. A model that is two points better on grounding and four times the price per answer is a real choice somebody should make deliberately. Which is the model selection question that Q28 works through for the narrow tasks where the cheap option usually wins.

Where this stops

Per answer pricing assumes the answer is the unit of value, and for a lot of AI features it is not.

An exhibitor summary has an obvious unit. A question-and-answer box over the warehouse does not, because the value of the eleventh question a show director asks on a Monday depends entirely on whether the first ten led her to it. Price that per active user per month and accept that the per answer figure is an operating metric with no revenue attached.

The second limit is that unit prices move, and they have generally moved down. An arithmetic exercise you did in March may be materially wrong by September, in your favour, and a feature you declined on cost grounds is worth re-running rather than remembering as settled. What does not move down is the salaried minute, so any conclusion resting on the human line is more durable than any conclusion resting on the token line.

The step this week is one query. Pull your model provider's usage log for the last full week, count completions, and divide by the number of AI outputs that actually reached a person in that period. If the ratio is above 1.5, find out what is being generated and thrown away before you spend any time optimising a prompt. Keep the cache decision in Q26 in view while you do it, because a good deal of that waste is regeneration of text that never changed. The wider set of platform decisions around AI sits with the pillar.