The clerical review band and how wide you can afford to make it

A portfolio load finished on a Friday and the review queue opened on Monday holding 11,300 pairs. Nobody had planned for 11,300. The previous run had produced a few hundred, and in between somebody had moved the lower cut off down by five points to catch more of the repeat attendees that the commercial team kept asking about.

That is the whole mechanism of the clerical review band. Its width is set in points of score, and it is paid for in human decisions, and the exchange rate between the two is a property of your score distribution that almost nobody measures before moving the number.

What sits between the two cut offs

The band is the middle of the three way decision that Fellegi and Sunter derived in 1969. Above the upper cut off a pair is a link, below the lower cut off a non link, and between them the pair is what they called a possible link, held rather than decided. Winkler's 2006 overview for the US Census Bureau names the same region plainly: "The region Tλ ≤ R ≤ Tμ is referred to as the no-decision region or clerical review region."

The three way structure is the reason the model can be honest. A two way rule forces every pair to a decision, including the ones where the evidence genuinely does not support one, and it does so silently. The band makes the ambiguity visible and hands it to somebody who can look at both records.

Making it visible also makes it somebody's Monday. The choice of the two numbers belongs to match threshold selection in J23. What follows here is the consequence of the gap between them.

Width is a count, and the count comes from your own file

Take a matching run over a portfolio file that produces 400,000 candidate pairs after blocking. Suppose the pipeline is configured with an upper cut off of 92 and a lower one of 75, purely as an illustration, and score every pair.

The distribution decides everything. In this run 6,900 pairs score 92 or above and merge automatically. Between 75 and 92 sit 4,800 pairs, which is 1.2 per cent of the candidate set. Everything below 75 is left alone.

Four thousand eight hundred decisions is the commitment you have made. Most of them will end in a rejection, so what you have signed up for is four thousand eight hundred occasions on which somebody opens two records and chooses. Whether that is a fortnight of work or a quarter depends on throughput, which is a number you have to measure on your own team rather than assume, and steward review capacity in K31 works that arithmetic properly. At an illustrative 40 decisions an hour it is 120 hours.

The important property is that none of this is proportional. Scores cluster. The gap between 75 and 92 might hold 1.2 per cent of pairs on one show and 4 per cent on another with a weaker set of identifying fields, and the same configuration line then means a fortnight in one place and two months in another.

What does one point of width cost?

The useful question is marginal. What happens if the lower cut off moves from 75 to 70?

In the same run, that slice holds 2,900 additional pairs, taking the band to 7,700 and 1.9 per cent of the candidate set. To find out what those 2,900 are worth, take 200 of them at random and label them by hand. Suppose 24 come back as true matches, so the slice is 12 per cent productive, and 2,900 times 0.12 gives roughly 348 real merges hiding in it.

Now the exchange rate is visible. Recovering 348 merges costs 2,900 decisions, which is 8.3 decisions per merge recovered. At that same illustrative 40 an hour, it is 72.5 hours of review for 348 merges, about 12.5 minutes of somebody's attention for each buyer correctly reunited with their history.

Compare that with the top of the band. Sample the 85 to 92 slice as well, and suppose it comes back 60 per cent productive, so the same effort there recovers five times as many merges. The band is not uniform, and the yield falls off steeply as you go down, which is exactly what you would expect from a score that is doing its job.

That gives you a rule that is easy to apply and hard to argue with. Widen downward while the marginal slice still pays for itself against work you have people to do, and stop where it does not. Publish the yield rate for each slice so the next person to move the number can see what they are buying.

A published figure for what review costs at scale

The event industry does not publish clerical review volumes, and the statistical agencies do, which makes their numbers the ones worth knowing.

Winkler's 2006 overview reports that in the 1992 US Census of Agriculture the computerised procedures "were able to reduce the clerical review from 75 clerks over 3 months to 6500 hours of review". Take a nominal 40 hour week over thirteen weeks and 75 clerks come to 39,000 hours, so the reduction is to roughly one sixth of the original effort. He adds that a validation follow up found the computerised procedures identified more duplicates automatically than the clerks had in the previous cycle, and that duplication in the final file fell to 2 per cent from 10 per cent five years earlier.

Two things are worth taking from that. Clerical review at national scale is measured in thousands of hours, so an event portfolio queue of a few thousand pairs is small by comparison and entirely tractable. And better automatic matching reduces the band rather than the review rate, which is the lever with the better return.

How wide can you afford to make it?

Start from the capacity you have and work back towards the score.

Decide how many decisions your team can absorb between the load and the deadline that matters, usually a mailing, a badge print or a board report. Multiply your measured throughput by the protected hours actually available, and you have a number of pairs. Then set the lower cut off wherever your score distribution puts that many pairs into the band, and record both the cut off and the count.

Doing it in this order has an effect worth naming. It converts an argument about a config value into an argument about staffing, which is a conversation the business already knows how to have. Somebody can then decide to fund a second reviewer for two weeks, or to accept a narrower band and a known number of unresolved duplicates.

When the band is still too big after that, three moves are open. Raise the lower cut off and write down the estimated merges given up, using the slice yield you already measured. Sample the band instead of working all of it, label a few hundred pairs, and apply the resulting rate as an adjustment to the duplicate count rather than pretending the pairs were resolved. Or improve the evidence, since a pair that scores 78 because the phone field was never parsed is a pair that would have scored 94 with two hours of normalisation work applied to the whole file.

The band drifts even when nobody touches it

A fixed pair of cut offs does not give you a fixed workload, and the queue that held 4,800 pairs in March can hold 9,000 in September with the configuration untouched.

Three things move it. The candidate set grows as editions accumulate, and pairs grow faster than rows. Field completeness changes when a show switches registration platform or drops a question from the form, and every field that stops arriving pushes a group of pairs down out of the automatic range and into the band. And the mix of audience shifts, so a year with a large international contingent produces more near misses on name and address than a domestic year does.

The cheap defence is a standing count. Log the size of the band and of each five point slice inside it at every load, next to the date and the row count, and treat a large move as an event that needs an explanation. A band that doubles between two loads is telling you something happened upstream, usually a feed that lost a column, and the queue length is often the first place it shows.

That count also gives the threshold conversation a history. When somebody proposes moving a cut off, the record of what previous moves did to the queue is more persuasive than any argument about score, and it complements the calibration question in J23 without repeating it.

Where this stops

The band is where the model is least able to help, which means it is also where human judgement is least reliable. Those two facts are the same fact seen from either side.

A reviewer looking at two rows with a similar name, no shared email and different employers is making the same inference from the same evidence the model had, with better context and worse consistency. Two reviewers working the same pairs will disagree on a meaningful share of them, and the only way to know your rate is to double key a sample and count. Do that once a quarter with 100 pairs given to two people, and treat the disagreement rate as a property of the queue rather than a fault in either reviewer.

Reviewer decisions are also training data, whether or not you use them that way, which is labelling training pairs in J36. Capturing them with the score, the field comparisons and the reviewer identity costs nothing at the time and is impossible to reconstruct later.

The limit worth admitting is that a wide band can be a way of avoiding a decision. If the queue is large because the score cannot separate your population, more human hours will not fix that, and the honest response is better fields or a better model, since a bigger queue spreads the same ambiguity across more people.

This week, take your last scored output, sort it by score, and count the pairs in each five point slice from your upper cut off down. That histogram costs one query and tells you what any future change to either number will cost in decisions, which is the piece of the unified data picture that usually turns up as a surprise on a Monday.