Match threshold selection and what each point of score actually costs you

Open the configuration for any matching pipeline that has been running for a year and you will find a threshold. Ask where the number came from and the answer is usually that somebody tried a few, looked at the merges, and stopped when the output seemed sensible. That is match threshold selection as most teams practise it, and the reason it feels uncomfortable is that nobody can say what the number buys or what it costs.

It can be done properly in about a day, and the day is mostly labelling.

Write the error budget before you look at any scores

The two errors are not symmetric in an event business, and the threshold conversation goes nowhere until somebody says so out loud.

A false merge takes two buyers and makes them one. One of them stops receiving marketing, disappears from the audience count and cannot be recovered by anyone who does not already know the merge happened. A false non match leaves a duplicate: visible, irritating, and fixable next time somebody looks.

So write the budget as a sentence with a number in it. At most one wrong merge in every 200 automatic merges, which is a precision of 0.995. Or one in 100, which is 0.99. The number is a commercial decision and it belongs to whoever owns the consequences, usually the person who signs off the audience figure. Once it exists, threshold selection stops being a matter of taste.

Where do the two cut offs come from?

Fellegi and Sunter set out the answer in the Journal of the American Statistical Association in 1969, and the model itself in J5 covers the derivation. Their decision rule splits pairs into three groups: links, non links, and a middle group they called possible links, where the decision is deferred to a person. They then proved a result about the ordering, that for stated bounds on the two error rates their rule minimises the size of that middle group.

The direction of the derivation is the part worth holding on to. You state the errors you will tolerate, and the cut offs follow. Winkler, writing for the US Census Bureau in 2006, puts it in one line: "The cutoff thresholds Tμ and Tλ are determined by a priori error bounds on false matches and false nonmatches."

Reading that backwards, picking the cut off and discovering the error rate afterwards, is the common practice and it inverts the logic. It also makes the number impossible to defend, because there is no stated standard it was chosen to meet.

Scoring a thousand labelled pairs by hand

The measurement is more ordinary than it sounds. Take candidate pairs from your own file, sample across the whole range of scores rather than only from the top, and have somebody decide match or not for each one. A thousand pairs is a comfortable size and a few hundred will do.

Suppose the sample holds 1,000 labelled pairs and 220 of them are true matches. Your matcher emits a match probability, so tabulate what happens at three candidate cut offs.

At 0.99, the model predicts 180 matches, of which 179 are true. Precision is 179 divided by 180, or 0.994. Recall is 179 divided by 220, or 0.814.

At 0.95, it predicts 205, of which 201 are true. Precision is 0.980 and recall is 0.914.

At 0.90, it predicts 240, of which 214 are true. Precision falls to 0.892 and recall rises to 0.973.

The Splink documentation, published by the UK Ministry of Justice for its open source linkage library, defines both measures the way they are used here: precision as the proportion of predicted matches that are genuine, recall as the proportion of actual matches the model found. It also makes the point that matters for this exercise, that "A model cannot be meaningfully summarised by just one of these performance measures".

Against a budget of 0.99 precision, only the first cut off qualifies. The automatic merge threshold is 0.99, and it costs you 41 true matches out of 220 that will have to be found some other way.

What does one slice of score actually buy?

The three cut offs above answer the question at three points. The more useful reading is the difference between them, because that is what you are actually deciding.

Dropping the cut off from 0.99 to 0.95 adds 25 pairs. Of those, 22 are true matches and 3 are wrong. That slice is 88 per cent correct, which is far too dirty to merge automatically against a 99 per cent budget and far too rich to throw away.

Dropping again from 0.95 to 0.90 adds 35 pairs, of which 13 are true and 22 are wrong. That slice is 37 per cent correct. A reviewer working it will spend most of their time saying no, and the yield is still worth having in a portfolio where a merged buyer is worth a rebooking conversation.

Keep going and the arithmetic collapses. Somewhere below that, a slice will hold 300 pairs with 4 true matches in it, and at 1.3 per cent correct a reviewer is doing 75 units of work for one result. That is the point where the lower cut off belongs, and the width of everything above it is the subject of the clerical review band in J24, which turns these counts into hours.

Two things follow from doing it this way. The cut offs are now defensible, because each one is the answer to a stated question. And the pairs between them are a known quantity rather than a surprise that appears in a queue.

The 41 matches you gave up are not gone

Setting the upper cut off at 0.99 left 41 true matches out of 220 unmerged, which is 19 per cent of the real links in the sample. That number reads badly on a slide and it is the correct outcome, given the budget. What matters is what happens to those pairs next, and there are three honest options.

Send them to a person. Most of the 41 sit in the slices immediately below the cut off, and a reviewer working the top of that range resolves them quickly because the evidence is nearly sufficient.

Get more evidence. A pair that scores 0.93 on name, employer and city will often clear 0.99 once a phone number or a postcode is parsed and added to the comparison. Missing fields, rather than a mis-set threshold, are behind a large share of near misses, and the fix improves every future run instead of one merge.

Accept the loss and report it. If the pairs are genuinely ambiguous and nobody has capacity to review them, the duplicate stays and the audience count is slightly high. Write down the estimated rate from your labelled sample, publish it next to the number, and let readers judge. A match rate quoted with its measured recall is worth more than a higher one quoted alone.

What you should avoid is quietly lowering the cut off until the 41 disappear, since they do not vanish. They become merges you have not measured, and the errors travelling with them are the expensive kind.

Thresholds do not transfer between shows

A cut off is a property of a scoring model applied to a particular file, and both halves of that change across a portfolio.

A show with a heavily corporate audience has email domains that identify employers, so agreements carry more weight and scores separate cleanly. A consumer facing show where two thirds of registrations use free mailbox providers loses that discrimination, the score distribution compresses, and the same numeric cut off sits in a completely different place relative to the two populations.

File size moves it as well. Doubling the file more than doubles the candidate pairs a matcher has to judge, nearly all of the additions are non matches, and the count of false merges at a fixed cut off rises with them even when precision at that cut off barely moves. If your portfolio runs eight shows, expect to calibrate per show, or at least per audience type, and to recalibrate when a show changes registration platform. The weights underneath the score come from the data too, which match weight calculation in J6 sets out.

Where this stops

The whole method rests on the labels, and the labels are the weakest part of it.

Start with the arithmetic on the sample above. Precision at the 0.99 cut off was 179 correct out of 180, which reads as 0.994. The standard error on a proportion that size is the square root of 0.994 times 0.006 divided by 180, which is about 0.006, so two standard errors is roughly one and a bit percentage points. That interval covers 0.98. On 180 pairs you cannot honestly distinguish a precision of 0.994 from one of 0.98, and anyone who quotes the third decimal place is reporting noise. Label more pairs if the distinction matters, or state the interval alongside the estimate.

Then there is what the labeller can actually know. A reviewer looking at two registration rows with different emails, different employers and a similar name is guessing, in the same way the model is guessing, and their guess becomes ground truth for a calibration exercise that then reports itself as accurate. Where you can, resolve a subset of hard pairs against something external, such as a lead capture record that carries both addresses or a delegate who confirmed at the desk, and keep those as the reference set.

The honest summary is that this procedure gives you a defensible threshold under your own labelling, which is a smaller claim than an accurate one, and still very much better than a number somebody picked in a sprint.

Take 300 candidate pairs from your last matching run this week, sampled across the score range rather than off the top, and label them by hand. Compute precision at three cut offs. You will either find that your current threshold meets the budget nobody wrote down, or you will find out what it has quietly been costing, and either outcome is worth the morning it takes before the next unified data decision gets made on top of it.