Deciding where AI is the wrong tool for an event data problem

A team spends six weeks building a model to decide which registrations need manual approval. The rules it ends up approximating are in a Word document that the operations manager wrote in 2019 and updates twice a year. Working out where AI is the wrong tool would have taken an hour at the start, and the hour was skipped because the question felt like an insult to the project.

Most of the AI failures I have watched in event businesses were scope errors. The model worked. It was pointed at a problem that did not need one, and the cost showed up later as maintenance nobody had budgeted for.

Three questions, asked before anyone writes a prompt

Ask them in this order and score each from zero to two.

How much judgement does the rule need? Zero means the rule is written down and somebody owns it. One means it is written down and has exceptions people argue about. Two means competent colleagues would decide differently and both be defensible.

How unstructured is the input? Zero means a database row. One means a form with predictable fields in an unpredictable layout. Two means free text, a scanned PDF, or a recording.

How tolerable is an error? Zero means an error costs money or turns away a customer immediately. One means it is caught by a review step before anyone outside sees it. Two means a wrong answer is a mild annoyance to one internal person.

Sum the three. Zero to one belongs in ordinary code. Four to six is a reasonable candidate for a model. Two and three are the interesting middle, where the answer depends on volume and on whether you can afford the review.

The scoring is arbitrary in its numbers and useful in its structure, because it forces three separate conversations that normally happen as one vague one about whether AI could help.

Is the rule stable, or does somebody just wish it were?

The first dimension is the one people score wrongly, in both directions, and it is worth slowing down on.

Teams overstate instability because the rule has never been written down. Registration approval feels like judgement right up until you sit with the person doing it and discover they apply six tests in a fixed order, and the sixth is "if in doubt, ask Priya". That is a rule with an escalation path, and it takes an afternoon to write out. The judgement score is zero and the escalation stays.

Teams also understate instability, usually in pricing and in anything touching sponsorship. A rate card looks like a table and behaves like a negotiation, with exceptions granted by whoever answered the phone. Writing that down produces a document that is wrong within a month, and a model trained on the historical decisions will faithfully learn the inconsistency.

The test that separates the two is cheap. Ask the person who owns the process to describe the rule, write it down as they speak, then take ten recent real cases and apply the written rule to each without them. Count how many match what actually happened. Nine or ten out of ten and you have a stable rule with a couple of exceptions. Five or six and the judgement is real, and it is worth knowing that before anyone builds anything.

Scoring four real jobs against the test

Badge validation at the hall door. Judgement: zero, because the entitlement matrix is a table with a version number and the operations manager owns it. Input structure: zero, because it is a badge record with a registration type and an event identifier. Error tolerance: zero, because turning away a paid delegate produces a complaint at the door and admitting an unentitled one is lost revenue. Total zero. This is deterministic code with a test per rule, and any model in that path is adding failure modes for nothing.

The narrative section of a post-show report. Judgement: two, since what counts as worth mentioning genuinely varies by reader and by year. Input structure: zero, because the input should be a list of named aggregates. Error tolerance: one, because a named person reviews the report before it leaves the building. Total three. Model with a review gate, and the aggregates computed upstream rather than by the model, for the reasons in why a model gets a percentage wrong.

Exhibitor contract field extraction. Judgement: one, because the fields are defined and the documents come in forty layouts. Input structure: two, a signed PDF. Error tolerance: one, since a wrong contracted stand size is expensive and is caught at invoicing. Total four. Model, with a confidence threshold and a review queue for anything below it.

Deciding which registrations look fraudulent. Judgement: two. Input structure: zero, since it is registration rows. Error tolerance: one. Total three, and the middle score is honest: this is a job for a scoring model over features rather than for a language model over prose, and the distinction matters because the two have different explainability properties.

Four jobs, four different answers, and none of them required a pilot to reach.

What the NIST framework puts before measurement

The ordering in the test above has an established analogue. The NIST AI Risk Management Framework, published as NIST AI 100-1 on 26 January 2023, organises its core into four functions: Govern, Map, Measure and Manage. Govern is described as "a cross-cutting function that is infused throughout AI risk management and enables the other functions of the process".

Map comes before Measure. Framing the context, identifying what the system is for and whether it is appropriate, precedes any attempt to quantify how well it performs. That ordering is the opposite of how most AI projects actually run, which is to build something, measure its accuracy, and reason backwards from the accuracy figure to whether the thing should exist.

An accuracy number cannot answer the appropriateness question. A fraud model at 94 per cent accuracy sounds excellent until you notice that 96 per cent of registrations are legitimate, at which point approving everything scores better.

The costs that only appear in year two

The initial build is the cheap part and the estimates people bring to these decisions almost never include the rest.

Sculley and colleagues catalogued the pattern at NIPS in 2015, naming "boundary erosion, entanglement, hidden feedback loops, undeclared consumers, data dependencies, configuration issues, changes in the external world, and a variety of system-level anti-patterns" as the risk factors that make machine learning systems expensive to keep running.

Three of those bite hard in event work.

Undeclared consumers: the assistant's output gets pasted into an exhibitor-facing report by somebody in sales, and now a tool built for internal exploration has an external audience nobody signed off.

Changes in the external world: your provider retires the model version, and the behaviour changes with no diff to review. Guarding against that needs a pinned question set re-run before every swap, which is ongoing work forever.

Hidden feedback loops: a lead scoring model ranks exhibitors, the sales team works the top of the list, those exhibitors renew, and next year's training data confirms the model was right. The loop closes and the model becomes unfalsifiable.

Against that, a written rule in code costs a review when the rule changes and nothing in between.

What if the answer has to be explainable?

This is the fourth question, and for an event business it is often the deciding one.

An exhibitor disputes their lead count. A sponsor asks why their category was ranked below another. A finance director asks how a renewal risk score was reached. In each case somebody has to give an answer that holds up in a commercial conversation, and "the model weighted these features" is a worse answer than "clause 4 of your contract sets the threshold at 200 scans".

Where the output touches a commercial relationship, the explainability requirement usually forces a rule regardless of how well a model performs. That is a business constraint rather than a technical one, and it does not get relaxed by a better model. Design for it at the start by asking who will dispute this output and what they will need to see.

Where this stops

The three-question test is a heuristic and it has a known failure. It scores the problem as currently framed, and problems get reframed. A job that scores zero today, because the rule is written down, scores differently the moment somebody decides the rule should adapt by segment. Rerun the test when the requirement changes rather than treating the original score as settled.

It also says nothing about volume. A judgement-heavy task over free text with tolerable errors scores four and still may not justify a model if it happens eleven times a year, because eleven manual decisions cost less than any system. Put the annual volume next to the score before anyone builds. Where the volume is high and the questions repeat, the answer is often neither a model nor a fresh rule but a stored query somebody already wrote, which is the argument in when rules beat a model for daily questions. The wider set of controls sits on the AI platform side.

Take the three AI projects currently proposed in your organisation and score each one this week, on paper, with the person who owns the underlying process in the room. The scoring takes twenty minutes per project. The conversations it forces are the ones that would otherwise happen in month four.