Skip to content

Auto approve threshold selection using a risk coverage curve on your own documents

OnboardingUpdated 2026-08-187 min read

In short

Auto approve threshold selection should come from a risk coverage curve built on your own labelled documents. Sort a labelled sample by confidence, accept the top share, and plot the error rate among accepted values against the share accepted. Read the threshold off the point where the error rate crosses the budget you are willing to carry.

Every document pipeline has a number in it that decides what gets accepted without a person looking. On most of them it is 0.95, or 0.90, or 0.8, and if you ask where it came from the answer is that it was in the example in the documentation.

Auto approve threshold selection is one of the few decisions in an onboarding pipeline that can be made empirically in a couple of days, on your own documents, with arithmetic anyone in the room can follow. The output is a defensible number per field and a written statement of what it costs you.

What is wrong with the default

A threshold copied from documentation encodes somebody else's document mix and somebody else's tolerance for error.

Two pipelines running identical software on identical thresholds can have wildly different accepted error rates, because the confidence distribution depends on your scans, your document layouts and your field list. A show taking clean broker-generated pdfs will have most of its mass above 0.98. A show taking phone photographs will have a long tail below 0.9, and the same cut-off accepts a completely different population.

The default also asserts a risk tolerance nobody agreed to. Setting 0.95 across every field says the show is equally relaxed about a wrong fax number and a wrong general liability limit, which no operations lead would say out loud.

Build the risk coverage curve from your own labelled sample

El-Yaniv and Wiener (2010) gave the trade-off its name. A classifier allowed to abstain buys accuracy on the cases it does answer by answering fewer of them, and they called the relationship between those two quantities the risk coverage trade-off. Coverage is the share of cases the system handles. Risk is the error rate among the ones it handled.

Auto approval is exactly that setting. Coverage is the share of field instances accepted with no human looking. Risk is the share of those accepted values that are wrong.

Building the curve takes four steps.

Take a labelled sample. Use the same held out gold set that produces your per field extraction scoreboard, which already carries a known correct answer for every field.

Record the confidence the pipeline assigned to each field instance, alongside whether the extracted value was right.

Sort descending by confidence, then walk down the list computing, at each point, how many instances you have accepted and how many of them were wrong.

Split the sample in half before you do any of this. Choose the threshold on the first half and measure the error rate on the second. A threshold chosen on the same data used to report its performance will always look better than it is.

The worked curve

Six hundred labelled field instances for one field, sorted by confidence, with invented figures so the arithmetic can be checked.

CoverageAcceptedErrors acceptedError rate
50 per cent30010.33 per cent
60 per cent36020.56 per cent
65 per cent39030.77 per cent
70 per cent42051.19 per cent
75 per cent45081.78 per cent
80 per cent480142.92 per cent
85 per cent510234.51 per cent
90 per cent540356.48 per cent
100 per cent6007813.00 per cent

Read the totals back in prose, because the shape carries the argument. Accepting the top 70 per cent of this field by confidence gives an error rate of 1.19 per cent among accepted values. Accepting the top 90 per cent gives 6.48 per cent. Accepting everything gives 13.00 per cent, which is simply the extraction error rate on this field with no threshold at all.

Where should the threshold actually sit?

Look at the marginal error rate, which is the error rate inside each additional slice rather than across the whole accepted set. That is where the curve tells you something the totals hide.

Going from 60 to 70 per cent coverage adds 60 instances and 2 errors, a marginal rate of 3.3 per cent. Going from 70 to 75 adds 60 instances and 3 errors, 5.0 per cent. Going from 75 to 80 adds 60 instances and 6 errors, 10.0 per cent. Going from 80 to 85 adds 9 errors, 15.0 per cent.

The marginal rate doubles between the 70 to 75 slice and the 75 to 80 slice. That doubling is the knee, and it sits at 75 per cent coverage. Read the confidence value of the instance at rank 450 in the sorted list, which in this example is 0.962, and that is your threshold.

Notice how far it sits from the default. A pipeline running 0.95 on this field would be accepting somewhere around 80 per cent of instances and carrying a 2.71 per cent error rate, roughly seventy per cent more errors than at the knee, and nobody would have known.

Choosing by error budget

The knee is a useful heuristic, and it is still only a shape on a curve. The better version is to state the error rate you are willing to carry on this field and read the coverage off the curve.

Say the field is a certificate holder name and the show decides one wrong value per hundred accepted is tolerable. That is a 1.00 per cent budget, which sits between the 70 per cent coverage reading of 0.95 per cent and the 75 per cent reading of 1.56 per cent.

Interpolate. Between rank 420 and rank 450 the errors rise from 4 to 7, so about 0.1 errors per additional instance. Accepting n instances gives 4 plus 0.1 times n minus 420 errors. Setting that over n equal to 0.01 gives 0.09n equal to 38, so n is about 422. Coverage is 422 over 600, or 70.4 per cent, and the threshold is the confidence at rank 422.

The budget for a general liability limit will be far tighter, perhaps one error in a thousand, and on this curve no coverage level reaches it, which is the correct and useful answer: that field cannot be auto approved on this extraction quality and every instance goes to a person. Deciding that field by field is the expected cost routing question, and the curve is what turns the decision into a number.

Does the threshold hold next month?

Only while the documents keep looking like the sample, and that is a real assumption rather than a formality.

Angelopoulos and Bates (2021) set out the distribution-free approach to this problem, where a threshold chosen on a calibration set carries an explicit, finite-sample guarantee that the accepted set contains the truth with a user-specified probability, without assuming anything about the model or the data distribution. Their framework requires exchangeability between the calibration data and what comes next, and that requirement is the practical warning for anyone setting a threshold on last season's documents.

Exchangeability breaks in ordinary ways in this business. A contractor redesigns the booth services order. A large exhibitor group moves to a European insurer with a different certificate layout. The portal starts accepting phone photographs. Any of those shifts the confidence distribution, and a threshold tuned on the old one silently accepts more errors while the reported coverage looks unchanged.

Two cheap defences. Track accepted coverage weekly, because a threshold that suddenly accepts 88 per cent where it used to accept 75 is telling you the input changed. And re-sample 50 labelled instances a quarter, run them through the current threshold, and check the accepted error rate against the budget you set.

Where this stops

A curve built on 600 instances is noisy exactly where you need it. At 75 per cent coverage the estimate rests on 7 errors, and the difference between 7 and 11 is one bad afternoon in the scanning room. Treat the knee as a region rather than a point, and prefer the conservative end of it, because the cost of accepting a wrong value is almost always higher than the cost of reviewing a right one.

The whole method also assumes confidence ranks correctly, meaning higher confidence really does mean more likely correct. Where that ordering is weak the curve will be flat and the knee will be an artefact of sampling. If your risk coverage curve rises smoothly with no knee at all, the honest reading is that this field's confidence carries little information, and no threshold on it will help.

And a threshold is only half a policy. The other half is what happens to everything below it, because a cut-off that routes 25 per cent of a high volume field to a queue nobody has staffed has moved the failure rather than removed it. Size the queue before you set the number.

Take one field from your held out sample, sort it by confidence, and compute the accepted error rate at 70, 80 and 90 per cent coverage. Three readings is enough to tell you whether your current onboarding threshold is anywhere near where your own data would put it, and it is about an hour of work in a spreadsheet. If the answer surprises you, run the same three readings on the field where a wrong value costs you the most, and start the straight through processing conversation from there.

Questions people ask about auto approve threshold selection

What is a risk coverage curve?
A plot of the error rate among accepted predictions against the fraction of cases accepted. Sorting by confidence and accepting from the top produces one point per coverage level. El-Yaniv and Wiener named this the risk coverage trade-off in 2010, and it is the standard way to reason about a system that can decline to answer.
How large a sample is needed to set a threshold?
Several hundred labelled field instances per field type, split in half. Choose the threshold on one half and measure the resulting error rate on the other. With fewer than about two hundred, the curve is too noisy at the top end, which is exactly the region the threshold sits in.
Should the same threshold apply to every field?
No. Each field has its own confidence distribution and its own cost of being wrong, so each needs its own curve and its own cut-off. A single threshold across all fields will be far too loose for the two or three that matter and needlessly tight for the rest.

Related reading

All onboarding operations articles