Skip to content

Human in the loop review for exhibitor documents without reviewing everything twice

OnboardingUpdated 2026-08-187 min read

In short

Human in the loop review works best when routing is decided per field by expected cost. Compare the cost of a coordinator checking a value against the cost of that value being wrong, and set the acceptance threshold at one minus the ratio between them. High stakes fields then route readily and low stakes fields almost never do.

The onboarding automation went live in March and by June two coordinators are opening every extracted document to confirm it. The typing has gone. The reading has not, and the reading was always most of the work. Somebody points out that the team is now doing the same job with an extra screen in it.

Human in the loop review is the right pattern and it gets implemented in the wrong shape. The usual shape is one confidence threshold applied to whole documents, which routes on how hard the document was to read and ignores entirely what the values in it are worth.

Microsoft (2026) recommends adding a human review stage for critical automation workflows, and it is worth being precise about the word critical. Criticality is a property of the individual field. A certificate of insurance holds one value that can stop a stand going up and six that cannot, and a review policy unable to express that difference will end up either too expensive or too dangerous.

Route on expected cost

The decision for each extracted field is a comparison between two quantities you can estimate in an afternoon.

The cost of review is what it takes a coordinator to check one field: open the document, find the value on the page, compare, accept or correct. Time a few and the answer for exhibitor paperwork usually lands around 90 seconds, call it 1.5 minutes.

The cost of an error is what happens if that field is wrong and nobody notices. Measure it in the same unit, coordinator minutes, so the two are comparable without inventing money.

Route to review when the probability the value is wrong, multiplied by the cost of it being wrong, exceeds the cost of checking. Rearranged, the acceptance threshold for a field is one minus the review cost divided by the error cost.

Mozannar and Sontag (2020) work the general version of this problem, learning a predictor that can either predict or defer the decision to a downstream expert, with the classifier and the rejector estimated together. Their point that the deferral rule belongs inside the system's objective is the one that transfers directly to onboarding: the question is where a coordinator adds more value than the model, and that depends on consequence as much as on uncertainty.

The threshold formula, worked per field

Take four fields from exhibitor paperwork and put a cost on each error, in coordinator minutes, with figures invented so the arithmetic is followable.

A wrong general liability each occurrence limit lets an underinsured stand onto the floor. If it surfaces at move in, it costs the operations team a day of firefighting, a hold on the build, and a conversation with the venue. Put it at 3,400 minutes. The threshold is 1 minus 1.5 over 3,400, which is 0.99956.

A wrong contracted booth area sends a stand to a space that does not fit, forcing a floor plan rework and a rebuild of the neighbouring aisle. Put it at 850 minutes. The threshold is 1 minus 1.5 over 850, which is 0.99824.

A wrong certificate holder name gets caught later and costs a phone call and a reissue request. Put it at 20 minutes. The threshold is 1 minus 1.5 over 20, which is 0.925.

A wrong producer fax number costs nothing, because nobody has sent a fax since 2011. The formula returns a negative threshold, which reads correctly as never review this field.

Four thresholds on one document: 0.9996, 0.9982, 0.925 and never. The first two are so close to one that the honest translation is review every one of them. That conclusion is uncomfortable and it is the right one, because a check that takes 90 seconds against an error that costs 3,400 minutes is worth doing 2,266 times over.

What does that do to the review queue?

It makes it bigger in volume and much smaller in risk, which is the opposite of what most teams expect.

Take 388 certificates of insurance in a month with eight extracted fields each, which is 3,104 field instances. Under a flat 0.95 threshold, suppose 18 per cent of instances fall below it. That routes 559 fields, at 1.5 minutes each, or 14.0 hours of review.

Under the per field thresholds above, the two liability limits route almost always, which is 776 fields. The additional insured presence check routes on a middling threshold, say 90 fields. The five cheap fields route almost never, say 30 between them. That is 896 fields, 22.4 hours, an extra 8.4 hours a month.

Now count what escapes. Under the flat threshold the accepted pile still contains roughly 2,545 field instances, and if the average residual error probability among them is around 1.2 per cent, about 31 wrong values are accepted. A quarter of those sit on the two liability limits, so roughly 8 underinsured stands a month go through unnoticed. Under the cost based thresholds that figure is closer to zero, because those fields are all being read.

Eight and a half hours a month to remove eight potential denied move ins is a trade nobody sensible refuses, and the flat threshold made it invisible. The overall share of documents clearing with no touch at all falls, which is why straight through processing rate has to be read next to an error rate rather than maximised on its own.

Design the review screen around the field

The routing decides what a person sees. The screen decides how long they take, and the 1.5 minute figure in the arithmetic above is something the screen design largely determines.

Show the cropped region of the page next to the extracted value, so the reviewer compares two things on one screen without scrolling a pdf. Show the confidence, both the field score and the worst word score, because what those two numbers mean together tells a reviewer whether to check the position or the characters.

Default the cursor into the value box and accept keyboard confirmation. A queue worked with one hand on the return key runs at a different speed from one worked with a mouse.

Show only the routed fields. A review screen that displays all eight fields when two were routed invites the reviewer to check all eight, which is how a per field policy quietly turns back into full document review within a month.

Why should every override be logged?

Because the overrides are the only unbiased error data your pipeline will ever produce, and they arrive free.

Log the document, the field, the extracted value, the confidence at extraction, the corrected value, the reviewer and the timestamp. Two questions become answerable from that table alone.

The first is whether a threshold is right. If reviewers accept 96 per cent of what they are shown on a field without changing it, the threshold on that field is too tight and you are paying for confirmations. If they change more than a third, it is too loose or the field is genuinely hard.

The second is where the systematic errors are. Overrides cluster. Fifty corrections on one field, all changing a date from an American reading to a European one, is a parsing rule rather than a training problem, and it will never show up in an aggregate accuracy number.

Retune monthly, in one sitting, using the previous month's override log. Move one threshold at a time and write down why, because a policy that changes for unrecorded reasons cannot be defended when somebody asks in November why an uninsured stand got through in September.

Where this stops

The formula assumes the confidence score is a probability you can multiply by a cost. On an uncalibrated model it is a ranking and nothing more, and multiplying a ranking by 3,400 minutes produces a number with no meaning. Everything here depends on that assumption being checked rather than assumed, and checking it is its own piece of work.

The error costs are also estimates made by the people who own the process, and they will be wrong in a predictable direction. Teams underestimate the cost of errors they have never seen and overestimate the cost of the last bad thing that happened to them. Revisit the cost table once a year with the incident log open, and expect at least one number to move by an order of magnitude.

There is a human limit as well. A reviewer working a queue of near-identical certificates will start accepting without reading somewhere in the second hour, and the measured accuracy of a human review stage falls through the afternoon. Cap the queue length per session, mix document types deliberately, and sample the reviewers themselves by planting a small number of known-wrong values and counting how many get caught.

Take one document type, list its fields, and write two numbers beside each one: minutes to check, and minutes lost if it is wrong and nobody notices. The thresholds fall out of that table arithmetically, and most teams find at least one field they have been reviewing for years that should never have been routed at all. The cut-off for the fields that do get routed can then be set empirically from your own labelled sample, which is what a risk coverage curve is for, and the whole policy sits inside the wider onboarding workflow rather than beside it.

Questions people ask about human in the loop review

Should every extracted document be reviewed by a person?
No, and a pipeline that reviews everything has automated the typing while leaving the reading, which is where most of the cost sits. Review the fields where a wrong value is expensive and accept the rest. The decision belongs at field level, because a single document usually contains both kinds.
How do you set a review threshold for a field?
Estimate the cost of a coordinator checking the field once, in minutes, and the cost of that field being wrong and going undetected, in the same unit. The threshold is one minus the first divided by the second. A cheap check against an expensive error gives a threshold close to one, which means review nearly everything.
What should be logged when a reviewer changes a value?
The document, the field, the extracted value, the confidence, the corrected value, the reviewer and the timestamp. That record is the only unbiased source of error data your pipeline generates, and it is what tells you next month whether a threshold is too tight, too loose or aimed at the wrong field entirely.

Related reading

All onboarding operations articles