Match weight calculation from m and u probabilities on badge scan data

An exhibitor sends you their lead retrieval export and asks how many of the 812 badges they scanned belong to buyers who registered as senior decision makers. The scan rows hold a name, a company, sometimes an email address, and a badge identifier that stopped being unique when the reprint desk opened on day two. Match weight calculation is how you turn each field comparison between a scan row and a registration row into a number of bits of evidence, and then add them up.

Two probabilities do all the work. The m probability is the chance that a field agrees given that the pair of records really is the same person. The u probability is the chance that the same field agrees given that the pair is two different people. The weight for an agreement is the base two logarithm of m divided by u, and the weight for a disagreement is the base two logarithm of one minus m divided by one minus u.

What is an email agreement actually worth?

Take email on badge scan data. Among pairs that really are the same person, the email agrees about 90 times in 100, because the scan captured whatever the registration held and the failures are mostly missing rather than wrong. So m is 0.90.

Among pairs that are two different people, agreement happens about once in a thousand, driven almost entirely by shared inboxes and by the handful of rows where an assistant registered several colleagues. So u is 0.001.

The agreement weight is log base two of 0.90 divided by 0.001, which is log base two of 900, or 9.81 bits. Each bit doubles the odds, so 9.81 bits multiplies the prior odds of a match by about 900.

Disagreement carries its own weight, and this is where intuition usually fails. Log base two of 0.10 divided by 0.999 is minus 3.32 bits. Email disagreement is evidence against the pair, and it is worth a third of what agreement is worth in the other direction. A rule based matcher treats a differing email as decisive. The arithmetic says it is a moderate negative, easily outweighed by two strong agreements elsewhere, which is exactly the behaviour you want when somebody changed employer between the registration and the scan.

The Splink documentation published by the UK Ministry of Justice states the same construction: the match weight is a measure of the relative size of m and u, expressed with base two logarithms, and the weights are additive across features. That additivity is the reason for the logarithm, and it is the property the whole model rests on.

Why the same surname is worth 6.8 bits or 12.1 bits

Aggregate weights hide most of the information in a name field, and this is the single cheapest improvement available to a working matcher.

Suppose surnames agree among true matches 93 times in 100. Computed in aggregate, with agreement between random pairs at 32 times in 10,000, the weight is log base two of 0.93 divided by 0.0032, which is 8.18 bits, and every surname in the file gets that same 8.18.

Now do it by value. The chance that two random rows both carry a particular surname is close to the square of that surname's frequency, so the u probability for a specific value is roughly the frequency itself once you condition on one row already holding it. For Smith at 0.85 per cent of your file, the weight becomes log base two of 0.93 divided by 0.0085, which is 6.77 bits. For Zielinski at 0.021 per cent, it becomes log base two of 0.93 divided by 0.00021, which is 12.11 bits.

Five and a third bits of difference between two agreements on the same field, which is a factor of about 40 in odds. Fellegi and Sunter built these value specific weights into the 1969 model, and Winkler, in his 2006 overview for the US Census Bureau, traces the idea back to Newcombe's work in 1959. It costs one extra table in your pipeline, holding each surname and its frequency, refreshed once an edition.

The same treatment applies to employer. Agreement on a company employing 400,000 people is weak evidence about two individuals. Agreement on a two person consultancy is strong. Anyone matching exhibitor contacts on company name alone is implicitly claiming those two agreements are worth the same.

Adding the fields up on one scan

Take a scan row and a registration row and work the total by hand.

• Email. Scan row has no email captured, registration row does, so this counts as a disagreement under a two level scheme: minus 3.32 bits.
• Surname. Both read Smith. Agreement, value specific: 6.77 bits.
• Company. Northgate Foods against Northgate Foods Ltd, token overlap above the agreement threshold, with m at 0.72 and u at 0.019: log base two of 37.9, or 5.24 bits.

The total is minus 3.32 plus 6.77 plus 5.24, which is 8.69 bits. That looks strong until you add the prior, which is the step teams skip.

Lead retrieval matching runs against a candidate set where roughly one pair in 1,200 is a true match, so the prior odds contribute log base two of 1 divided by 1,199, which is minus 10.23 bits. The posterior is 8.69 minus 10.23, or minus 1.54 bits, odds of about 0.34 to one, a probability near 0.26. A pair that felt convincing when you read the fields is more likely wrong than right once the size of the candidate set is accounted for. How the ranking and the prior fit together is the subject of probabilistic record linkage.

How many bits do you actually need?

Work backwards from the answer you want and the target becomes concrete.

With a prior of minus 10.23 bits, a pair needs 10.23 bits of field evidence just to reach even odds. To reach a posterior probability of 0.99, the odds have to be 99 to one, which is log base two of 99, or 6.63 bits above even. So the fields have to supply 16.86 bits in total.

An email agreement gives 9.81 of those. A rare surname gives 12.11 on its own and a common one gives 6.77. Company agreement adds 5.24. Two moderate agreements will not get you there against a candidate set of this size, and that is the honest reason lead retrieval matching needs either a stronger key or a much smaller candidate set. Tightening the blocking so that one pair in 60 is a true match moves the prior to minus 5.9 bits and drops the requirement to 12.5 bits, which two ordinary agreements can clear.

Where do the m probabilities come from?

The u probabilities are the easy half. Count value frequencies in your own file and you have them, because agreement between two randomly chosen rows is a property of the data you already hold. No labels required.

The m probabilities need something else, and there are three honest routes. Label pairs by hand and count how often each field agrees among the true ones, which is the most defensible and the slowest. Bootstrap from a field you trust: take pairs that agree exactly on email plus mobile, treat them as true matches, and measure how often surname and company agree within that set. Or estimate them from the score distribution itself with an expectation maximisation algorithm, an approach Winkler described in 1988 and reviewed in his 2006 overview, which fits a two class mixture to the observed agreement patterns without any labels at all.

I would use the bootstrap first and the algorithm second, and I would check both against 200 hand labelled pairs before anything merges. The failure mode of unsupervised estimation is quiet. It can settle on two classes that have nothing to do with matching, such as complete rows against sparse ones, and every number downstream then describes your form design.

Where this stops

Weights computed this way assume the fields are independent given match status, and registration data breaks that assumption in an obvious place. Employer name and work email domain carry the same fact, so adding both weights counts one piece of evidence twice and inflates the total. The correction is in the conditional independence assumption, and it usually costs a few bits on exactly the pairs you were most confident about.

The second limit is that a two level comparison, agree or disagree, throws away most of what a string comparator tells you. Kathryn against Katherine and Kathryn against Jones are both disagreement under this scheme, which is plainly wrong. The fix is three or four comparison levels per field, each with its own m and u, and it roughly doubles the estimation work.

The third is that missing is not disagreement, though the arithmetic above treated it that way for simplicity. A field absent from one row carries no evidence in either direction, and forcing it into the disagreement bucket penalises sparse rows systematically, which means the channels that collect fewer fields will look like they bring worse people. Give missing its own level with a weight near zero.

Start with one field this week. Pull the surname frequency table from your registration archive, find the ten most common values, and compute log base two of 0.93 divided by each frequency. The spread between the top and the bottom of that list is the evidence your current matcher is throwing away every time it treats a surname agreement as one thing, and it is a five minute argument for putting a frequency table into the unified data layer. Then read the Fellegi Sunter model for what to do with the totals once you have them.