Designing human in the loop approval that people actually read

A show director opens the approval queue on the Thursday before move-in. There are 340 exhibitor performance emails in it, and the queue is empty again nine minutes later. That works out at 1.6 seconds per email, which is enough time to move a mouse and press a key, and nowhere near enough time to read a paragraph containing six numbers. The human in the loop approval step existed, it was logged, and the audit trail now says a named person reviewed every one of them.

Nothing in that queue was checked. The screen made checking impossible and approving trivial, and people do the thing the screen makes easy.

What an overseer is actually supposed to be able to do

Article 14 of the EU AI Act, Regulation (EU) 2024/1689, is unusually concrete about this, and it is worth reading as an interface specification rather than as compliance text. The oversight measures have to let the person "correctly interpret the high-risk AI system's output, taking into account, for example, the interpretation tools and methods available", let them "decide, in any particular situation, not to use the high-risk AI system or to otherwise disregard, override or reverse the output", and let them "intervene in the operation of the high-risk AI system or interrupt the system through a 'stop' button or a similar procedure".

Three capabilities: interpret, override, stop. Whether your exhibitor reporting sits inside the Act's high-risk list is a separate question that belongs with Q36. Treat the three capabilities as good design regardless, because they describe the difference between a review and a signature.

Article 26 of the same Regulation adds the part organisations skip. Deployers "shall assign human oversight to natural persons who have the necessary competence, training and authority, as well as the necessary support". Authority matters more than it sounds. If the reviewer knows that rejecting a batch means a conversation with the person who built the pipeline, and approving it means going home, the approval rate tells you about the org chart.

Why does an approval screen get clicked through?

Because most of them present a finished paragraph and one green button, and a finished paragraph is genuinely hard to check.

Consider what verifying a single sentence costs. "Scan volume at your stand rose 18 per cent against last year, from 412 to 486." To confirm that, a reviewer needs to know which scan table produced 412, whether 412 covers the same three days as 486, whether the 2025 edition ran four days, and whether 18 per cent is 486 divided by 412 minus one. That last one they can do in their head in about four seconds. The first three require opening a different system, and at that point the marginal cost of care is several minutes per email against a queue of 340.

So the reviewer does the arithmetic they can do and trusts the arithmetic they cannot. Then a batch goes out where one show's figures were joined to another show's date range, and the error was in the part nobody could check from the screen.

This is not a character flaw in the reviewer, and it is worth knowing that the research treats it as an attentional effect rather than a motivational one. Parasuraman and Manzey, reviewing the empirical work in Human Factors in 2010, reported that automation complacency "occurs under conditions of multiple-task load, when manual tasks compete with the automated task for the operator's attention", and that it appears in expert and inexperienced participants alike. The Thursday before move-in is multiple-task load in its purest form. Measuring the effect and keeping approval rates honest is Q23's work; this post is about the screen that produces it.

The fix is to make the expensive checks cheap rather than to ask people to try harder. Everything else in this post follows from that.

Show the diff and the operands

Two design choices do most of the work.

The first is showing the source aggregates next to the prose. For the sentence above, the panel beside it carries scans_2026 486 from fact_badge_scan, days 3; scans_2025 412 from fact_badge_scan, days 3; delta 74; delta_pct 17.96. Now every claim in the sentence is checkable without leaving the page, the day-count mismatch would be visible as a difference between two integers, and the reviewer's four second mental division is the last thing they need to do rather than the only thing they can do.

The second is showing what changed. When a draft is a regeneration of something the reviewer has seen before, present it as a diff against the previous approved version rather than as fresh prose. A reviewer reading 340 near-identical exhibitor emails is doing an inefficient job on 340 documents. A reviewer reading 340 diffs is doing a precise job on the eleven that changed, and the other 329 can carry a one line statement that the text and every figure in it are unchanged since the approved version.

That is a real reduction in work. If eleven of 340 drafts differ from their approved predecessor, and a careful read of a changed draft takes ninety seconds, the whole queue is sixteen and a half minutes of actual reviewing plus a couple of minutes of confirming the unchanged set. Longer than nine minutes, and it is nine minutes of nothing against eighteen minutes of something.

A worked example: one exhibitor email on the screen

Take a single draft going to an exhibitor after a show closes.

The prose block says the stand recorded 486 badge scans across three days, up from 412 at the 2025 edition, that 63 per cent of scanned visitors held a buyer badge type against a show average of 58 per cent, and that the stand's busiest hour was 11:00 to 12:00 on day two with 71 scans.

Beside it sit six rows, one per figure, each with a name, a value, a period and a source table. Under those, one line of derived arithmetic the model was required to quote: 486 minus 412 is 74, and 74 divided by 412 is 17.96 per cent.

Three controls sit under that. Approve. Reject with a reason picked from a short list, where "figure looks wrong" routes the draft to the data owner rather than back to the queue. Edit, which opens the prose and records the change.

Now the timings. A reviewer who trusts the pipeline scans six aggregate rows and one arithmetic line in perhaps twenty seconds. A reviewer who spots that scans_2025 covers four days while scans_2026 covers three rejects in five seconds, because the day counts are two adjacent integers on the same panel. Neither of those is possible when the screen shows a paragraph and a button.

Note what the reject reason does. It turns the review into a routing decision, so a wrong number goes to whoever owns the number instead of dying in a comment box. That is also what makes the whole thing measurable, and measurement is the subject of keeping approval rates honest in Q23.

What belongs on an approval screen?

The panel should carry the aggregates a figure in the prose depends on, their periods, their source tables, and any derived arithmetic quoted with its operands. It should carry the version this draft is a change to, if there is one.

It should not carry a confidence score. A number between zero and one attached to a paragraph gives a reviewer something to defer to without giving them anything to check, and deferring is the failure mode you are designing against. If part of a draft rests on an estimate with an interval, show the interval in the prose where the reader will see it too.

It should not carry the full prompt. Reviewers who are shown the prompt start debugging the prompt, which is a different job done by a different person on a different day.

Make rejecting cheaper than approving

This is the part that changes behaviour, and it is mostly about defaults.

If approval is one click and rejection needs a typed explanation, you have priced honesty above convenience and you will get convenience. Give rejection a short reason list, let it be one click plus one click, and let the reason list be specific enough to be useful later: figure looks wrong, wrong tone for this account, missing context the exhibitor will ask about, do not send to this account at all.

Then let the batch fail loudly. If a reviewer rejects three drafts in a run of forty on the grounds that a figure looks wrong, the sensible default is to hold the remaining thirty-seven rather than send them, because a systematic upstream fault is far more likely than three independent coincidences. Holding the batch is also the "interrupt the system" capability Article 14 asks for, expressed as something an event operations lead would actually press. Where those holds belong in the wider workflow, and which actions need a gate in the first place, is the placement question that Q22 takes on.

Where this stops

An approval screen makes a draft checkable against its own inputs. It does nothing about whether those inputs are right.

If badge scan coverage at a hall entrance is 60 per cent and the pipeline treats scans as attendance, every figure on the panel traces cleanly to a table and the email is still wrong by 40 per cent. The reviewer will approve it, correctly, because the screen is answering the question it was built to answer. Provenance is a defence against invention and it is no defence at all against a measurement error at source.

There is a second limit that is more uncomfortable. A well-designed approval screen raises the cost of the review, and somebody has to pay for it in salaried minutes. Eighteen minutes on a batch of 340 is cheap. Eighteen minutes per batch across nine shows and a weekly cadence is a part-time job, and that arithmetic belongs in the same business case as the model spend, which is what the feature really costs in Q24. Pretending the review is free is how the review ends up at 1.6 seconds an item.

The step this week is a measurement rather than a build. Pull the approval log for whatever AI-assisted output your team already sends, and divide the elapsed time between the first and last approval in each batch by the number of items in it. If any batch comes out under five seconds an item, you have a signature step and no review step, and the screen is the reason. Read more on how these systems get built and governed before you redesign it.