Skip to content

Document type classification accuracy comes before any field on the page matters

OnboardingUpdated 2026-08-188 min read

In short

Document type classification accuracy is the share of uploads routed to the correct document type before any field is extracted. Measure it with a confusion matrix over roughly a hundred labelled examples per type. Where the errors concentrate in one pair of types the fix is specific and cheap, and where they scatter the model needs more data.

A coordinator flags that the insurance extraction has got worse. Certificates are coming through with no policy number, no limits and no expiry date, and the assumption in the room is that exhibitors are uploading bad scans again.

They are uploading W-9s. The portal's insurance slot accepts any pdf, exhibitors upload the tax form and the certificate as one combined file, and the classifier is reading page one. Every field on those documents was extracted correctly against entirely the wrong field list.

Document type classification accuracy is the measurement that catches this, and it sits upstream of everything else in an onboarding pipeline. Nobody reports it, because the classifier is usually a single line of configuration somebody set up in week one and never looked at again.

Classification happens first and it decides everything after it

An exhibitor onboarding pipeline typically handles five document types: the exhibitor contract, the certificate of insurance, the IRS W-9, the booth services order and the exhibitor profile form. Each has its own field list, its own validation rules and its own acceptance thresholds.

The classification step picks which of those five to apply. Get it right and the field extraction has a chance. Get it wrong and every subsequent number is meaningless, including the confidence scores, which will be low and will be blamed on image quality.

Harley, Ufkes and Derpanis (2015) built the document image classification benchmark that most of this work traces back to, releasing 400,000 labelled document images across 16 categories from the IIT-CDIP collection. Sixteen categories over four hundred thousand images is the scale at which this problem was studied, and your five categories over a few thousand documents a season is an easier problem in every respect except one: your five look far more like each other than a letter looks like a scientific paper.

Microsoft (2026) ties the same idea to a number it returns, describing document type confidence as an indicator of how closely an analysed document resembles the training set, with low scores pointing at template or structural variation. That score exists in most pipelines and almost nobody stores it.

How do you measure classification accuracy properly?

Take about a hundred real uploads per document type, drawn from a full onboarding window rather than from whatever folder is convenient. Label the true type by hand. Run them through classification. Cross-tabulate true type against predicted type.

A hundred per type is the smallest sample that makes the matrix readable. Below fifty, a cell containing four errors could be a systematic confusion or could be nothing, and you will not be able to tell.

Sample from real traffic, including the documents that arrive badly. If a fifth of your certificates come in as phone photographs, a fifth of the sample should be phone photographs. An evaluation built on clean uploads produces a flattering number that your live pipeline never reproduces.

The confusion matrix, worked

Five hundred documents, a hundred of each type, with invented figures so the reading is followable. Rows are the true type and columns are what the classifier predicted.

True typeContractCertificateW-9Services orderProfile form
Contract941032
Certificate of insurance291511
IRS W-91148311
Booth services order400888
Exhibitor profile form310987

Overall accuracy is the diagonal over the total: 94 plus 91 plus 83 plus 88 plus 87, which is 443 out of 500, or 88.6 per cent. That is the number that would go on a slide, and it is close to useless on its own.

Recall per type is the row: contracts 94 per cent, certificates 91, W-9s 83, services orders 88, profile forms 87. Precision per type is the column. The classifier predicted certificate 107 times and was right 91 of those, which is 85.0 per cent precision. It predicted W-9 88 times and was right 83 times, which is 94.3 per cent. Certificates have the best recall of the awkward types and the worst precision, and those two facts point in opposite directions.

What does the shape of the errors tell you?

There are 57 errors in the matrix. Where they sit matters more than how many there are.

Fourteen of the 57, which is 24.6 per cent of all errors, live in one cell: true W-9 predicted as certificate of insurance. Fourteen errors piled into a single cell is a pattern with a cause, and in practice the cause is almost always the combined upload from the opening paragraph, where the exhibitor scans the certificate and the W-9 as one file and the classifier reads the first page. The fix is page level splitting before classification, which is a day of work and removes a quarter of the errors in the pipeline.

Seventeen more errors sit in the services order and profile form pair, 8 one way and 9 the other, which is 29.8 per cent of the errors in two symmetrical cells. Symmetry means genuine visual similarity. Both documents are show branded, both open with a company details block, and the layouts are cousins. Training data will help here, and so will a cheaper trick: both forms carry a form identifier in the footer, and a text rule that reads it will beat a layout model on this pair.

The remaining 26 errors are spread thinly across the other cells, one and two at a time. That is the pattern that genuinely calls for more labelled examples, and it is the only one of the three where the answer is the expensive one.

An accuracy figure of 88.6 per cent is compatible with all three of those situations. The matrix separates them in about ten minutes.

What a misclassification costs downstream

Follow one misrouted W-9 through the pipeline and the cost becomes concrete.

The certificate extractor runs against it, finds no policy number, no general liability limit and no expiry date, and returns low confidence on every field because none of them are on the page. The document lands in the review queue looking like a bad scan. A coordinator opens it, sees a tax form, and either refiles it or emails the exhibitor asking for the certificate the exhibitor has already sent.

At a show taking 1,900 documents a season with roughly 22 per cent of them W-9s, that is about 418 W-9s. At 83 per cent recall, 71 of them go to the wrong extractor. If each one costs a coordinator six minutes of confusion and a follow up email, that is 426 minutes, or 7.1 hours, spent on a defect that reports itself as an extraction problem. The per field extraction scoreboard will look worse for reasons that have nothing to do with extraction, which is why classification has to be measured first and separately.

Let the classifier decline

The most useful configuration change in most pipelines is allowing a sixth outcome: unknown.

A classifier forced to choose among five types will always choose one, and it will choose confidently on documents that are none of them. Exhibitors upload plenty of those. Booth renderings, invoices from their stand builder, a photograph of a business card, last year's certificate.

Store the document type confidence, set a floor, and route everything below it to a person as unclassified. On the matrix above, that will catch a good share of the thinly scattered errors, because those documents usually score low. It will catch fewer of the W-9 and certificate confusions, because the classifier is confidently right about page one. Confidence gets you the random errors and does nothing about the systematic ones, which is a general property worth remembering when you read field level confidence scores.

Then measure the unknown bucket as its own class. Count how many documents land there each week and what a coordinator decides they were. A bucket running at 2 or 3 per cent of volume, mostly containing things that genuinely are none of your five types, is working correctly. A bucket at 15 per cent is a threshold set too high, and it has quietly converted an automation project into a manual sorting job with extra steps. A bucket at 0.2 per cent means the floor is doing nothing and every strange document is still being forced into one of the five.

The other reason to keep the bucket is that it is free research. Read what accumulates in it over one season and you will find the document types your portal should have had a slot for, which is usually two or three things exhibitors keep sending that nobody ever designed a place to receive.

Where this stops

A confusion matrix built once describes the pipeline once. Document layouts change every year: a contractor redesigns the services order, an insurer changes its certificate template, the show rebrands its own profile form. Rebuild the matrix each season on fresh uploads, and expect the shape of the errors to move even when the headline number does not.

The measurement also assumes one document type per file, and the opening example is the case where that assumption breaks. A file containing a contract, a certificate and a W-9 has no single correct label, and a classifier scored against a per file label will look wrong when it was doing its best. If combined uploads are more than a few per cent of your traffic, measure classification at page level and treat file splitting as its own step with its own accuracy.

And the matrix says nothing about the documents that never arrived. A type nobody uploads because the portal slot is hidden will show perfect accuracy on the handful you have, and the real problem is a completion problem rather than a classification one.

Pull a hundred uploads per document type from last season, label the true type in a spreadsheet, and cross-tabulate against what your pipeline decided. If a quarter of your errors turn out to be in one cell, you have found a day of work that fixes a quarter of the problem, and the rest of your onboarding automation reporting, including the share of documents processed with no human touch, gets more honest the moment classification is measured on its own.

Questions people ask about document type classification accuracy

Why measure classification separately from extraction?
Because a misclassified document produces field errors that look like extraction failures. A W-9 sent to the certificate of insurance extractor returns missing limits and a missing policy number, and a coordinator reads that as a poor scan. The underlying fault is upstream, and no amount of extraction tuning will touch it.
How many examples per document type are needed?
Around a hundred correctly labelled examples per type gives a confusion matrix you can read. Fewer than fifty and a single cell of three or four errors is indistinguishable from noise. Draw them from real uploads across a full onboarding window rather than from a folder of clean examples somebody kept.
What does a confusion matrix show that an accuracy figure does not?
Where the errors go. An overall accuracy of 88.6 per cent is consistent with errors spread evenly across every pair of types, or with a quarter of them sitting in a single cell. Those two situations need completely different work, and only the matrix distinguishes them.

Related reading

All onboarding operations articles