Measuring document extraction accuracy on exhibitor paperwork one field at a time

The vendor says 94 per cent. Your operations lead asks 94 per cent of what, and the answer takes three emails and still comes back as a document level figure computed over a sample nobody can see. Meanwhile a certificate of insurance came through last week with the general aggregate limit read as 2,000,000 when the certificate says 2,000, and nobody caught it until the fire marshal did.

Document extraction accuracy stated as one percentage is the least useful form the measurement can take, and it is the form almost everyone reports. The number that matters is per field, and the fields on an exhibitor certificate of insurance differ from each other by more than any two extraction systems differ from each other.

Jaume, Ekenel and Thiran (2019) built the FUNSD dataset from 199 real, fully annotated scanned forms precisely because form understanding needs labels at the level of the entity on the page. The same logic applies to your onboarding pipeline, and the work involved is a fortnight rather than a research programme.

Why does document level accuracy hide the problem?

Because it compounds, and because it flattens consequence.

Take a certificate of insurance with eight fields you actually use. Suppose every field is right 96 per cent of the time and errors are independent. The chance that all eight are right on a given document is 0.96 to the eighth power, which is 72.1 per cent. Report that as document accuracy and the system sounds broken. Report the same system as 96 per cent field accuracy and it sounds excellent. Both numbers describe identical output.

Run it the other way and the effect is worse. A vendor quoting 99 per cent per field on the same eight field document is quoting 92.3 per cent of documents fully correct, and a buyer hearing 99 per cent is not imagining one document in thirteen with something wrong in it.

Compounding is the smaller half of the problem. The larger half is that the eight fields do not matter equally. Getting the certificate holder's street address wrong costs a phone call. Getting the general liability each occurrence limit wrong puts an uninsured stand in your hall. An average over those two treats them as interchangeable, and the average will always look good, because most fields on most forms are printed cleanly in a box.

Build the gold set before you measure anything

Nothing here works without a held out set of documents with known correct answers, labelled by a person, kept separate from anything used to configure the system.

Two hundred documents per document type is a sensible target. That gives you enough examples of the awkward cases to see them, and it is small enough that one coordinator can label it across a few afternoons with a spreadsheet and a defined field list.

Stratify it against real traffic. If 30 per cent of your certificates arrive as photographs of a printed page taken on a phone, 30 per cent of the gold set should be photographs of a printed page taken on a phone. A gold set built from the clean documents that were easy to find will report accuracy that your live pipeline never reproduces, and this is the single most common way these evaluations go wrong.

Label the fields, and label the absences. A certificate with no additional insured wording has a correct answer of absent, and a system that invents wording on that document is making the most dangerous error in the whole pipeline. Absence is a value and it has to be in the gold set as one.

Hold the set out and mean it. Once anybody has tuned a rule against a document, that document has stopped being evidence.

Choose a match rule per field, and write it down

The match rule decides the score, sometimes more than the model does, and most teams never write theirs down.

Exact match after trimming suits a policy number. Strip leading and trailing whitespace, compare case insensitively, and anything else counts as wrong, because policy numbers have no legitimate variants.

Normalised match suits a legal entity name. Lowercase, strip punctuation, strip legal suffixes such as inc, llc, ltd and gmbh, collapse internal whitespace, then compare. Acme Retail Group, Inc. and ACME RETAIL GROUP INC are the same company, and any scoring rule that calls them different is measuring formatting.

Numeric comparison with a stated tolerance suits booth square feet, where within one unit is fine. Currency limits get zero tolerance, because 2,000 and 2,000,000 differ by exactly the thing you are checking.

Normalised date suits policy effective and expiry dates, parsed to an ISO date before comparison. Record the assumed locale, because 03/04/2026 is two different dates and your certificates arrive from both conventions.

Presence suits the additional insured wording, where the answer is present or absent and the extraction either found the clause or did not.

Microsoft (2026), documenting its own extraction service, keeps the same separation, distinguishing an estimated accuracy score produced when a model is trained from the confidence returned per field when a document is analysed. Two different quantities, reported in two different places, and mixing them up is how an operations team ends up quoting a training statistic as a live error rate.

Here is why the choice matters. On the same 200 certificates, scoring the named insured by raw exact match returns 143 correct, or 71.5 per cent. Scoring the identical output with the normalised rule returns 191, or 95.5 per cent. That is a 24 point swing produced entirely by a decision about scoring, with no change to the system at all. Publish the rule alongside the number or the number means nothing.

The worked scoreboard

Take the 200 document gold set for certificates of insurance and score the eight fields separately.

Named insured, normalised match, 191 correct, 95.5 per cent. Policy number, exact match, 197 correct, 98.5 per cent. Policy effective date, 196, or 98.0 per cent. Policy expiry date, 195, or 97.5 per cent. Certificate holder name, 188, or 94.0 per cent. General liability each occurrence limit, 184, or 92.0 per cent. General aggregate limit, 179, or 89.5 per cent. Additional insured wording present, 152, or 76.0 per cent.

Those eight sum to 1,482 correct out of 1,600 field instances, a mean field accuracy of 92.6 per cent. Measured directly, 138 of the 200 documents had all eight fields right, which is 69.0 per cent document accuracy.

One system now has three numbers against it. 92.6 per cent of field instances correct. 69.0 per cent of documents fully correct. And 76.0 per cent on the single field that decides whether an exhibitor is allowed to build. The third is the one your operations team needs, and it is invisible in either of the first two.

That ranking also tells you where to spend. The additional insured field is a clause buried in a free text remarks box, so the fix is a different extraction approach for that field rather than more training data for the whole form. The aggregate limit at 89.5 per cent is a currency parsing problem, and currency parsing problems are usually one afternoon of rules.

What should the accuracy number be weighted by?

By what the error costs you, which means writing down the cost before you look at the scores.

Give each field a consequence band. Blocking means a wrong value stops or should stop a stand going up, and that band holds the two liability limits, the additional insured wording and the policy expiry date. Correctable means a wrong value gets caught later at the price of a phone call, which covers the certificate holder name and the address. Cosmetic means nobody will ever notice, which covers the producer's fax number, still printed on the form in 2026.

Then report accuracy per band. On the numbers above, the blocking fields are the additional insured wording at 76.0 per cent, the each occurrence limit at 92.0, the aggregate at 89.5 and the expiry date at 97.5, which averages 88.8 per cent across the four. The correctable and cosmetic fields sit above 94 per cent and pull the headline up.

Report the blocking band as the headline and the rest as detail. It is a lower number and a harder conversation, and it is the one that corresponds to the risk the show is carrying. What the pipeline does with those scores, which documents it accepts without a human and which it routes, is decided by the confidence attached to each field rather than by the aggregate score.

Refreshing the gold set

A gold set decays. Insurance forms change, brokers change portals, a large exhibitor switches to a European insurer and a new layout arrives in volume.

Re-sample 50 documents a quarter from live traffic, label them, and add them. Retire the oldest 50 at the same time so the set stays around 200 and stays representative. Keep a permanent frozen subset of about 30 documents that never changes, so you have one series that is comparable across years even as the rest of the set moves.

Score the whole set on every material change to the pipeline, and record the score with the date, the model version and the match rules used. That record is what stops the argument about whether last month's change helped.

None of this measures whether you were reading the right document in the first place. A certificate scored perfectly against certificate fields, when the file was actually a W-9, scores zero and tells you nothing, which is why document type classification has to be measured first and separately.

Where this stops

A gold set of 200 documents gives you a confidence interval, and the interval is wider than most people assume. At 200 documents, a field measured at 95 per cent carries a 95 per cent interval about three points wide on each side, running from roughly 92 to 98 per cent. A change from 95.0 to 96.2 after a configuration tweak sits comfortably inside that, and treating it as a result will send a team chasing noise for a quarter. Either accept that you can only detect large changes, or build a bigger set for the two or three fields you care most about.

Per field scoring also says nothing about which documents fail together. If the same 12 badly scanned certificates account for most of the errors across every field, that is a single image quality problem showing up in eight columns. Look at the error rows before you plan the work.

And a gold set measures the system against a person's labels. Where the person was wrong, or where two coordinators would disagree about whether a remarks clause counts as additional insured wording, the ceiling on measurable accuracy is your own labelling agreement. Double label 30 documents, measure how often two people agree, and treat that figure as the ceiling.

Pick your highest volume document type, pull 200 recent examples from the last edition, and label just the fields in your blocking band. You will have a defensible per field scoreboard in about two days, and the onboarding decisions that follow from it, including what share of documents can safely be processed with no human at all, all read from that same table.