Probabilistic record linkage when your registration files share no common key

You have bought a show. The registration file that came with it holds 38,000 rows with forename, surname, company, job title and a postal address, exported from a platform that was retired in 2021. Your own file holds email addresses, mobile numbers and badge scans. The question from the integration lead is how many people appear in both, and there is no key to join on.

Probabilistic record linkage is the method for that situation. It scores partial agreement across several weak fields, gives every candidate pair a number, and lets you rank the pairs and cut the ranking where the evidence stops being good enough. No single field has to be decisive, which is the point, because in this file none of them is.

The theory is older than most of the software implementing it. Fellegi and Sunter set out the decision theory in the Journal of the American Statistical Association in 1969, building on Newcombe and colleagues in Science ten years earlier. What has changed since is that the arithmetic now runs over tens of millions of rows on hardware you already have.

Scoring one pair over four fields

Take two rows. Both say Anna Kowalczyk. Both give an outward postcode of M15. The email domains differ, because one row predates a job change. Forename and surname agree exactly after normalisation.

Each field needs two numbers. The first is the probability that the field agrees when the pair really is one person. The second is the probability that it agrees when the pair is two different people. Divide the first by the second and take the base two logarithm, and you have that field's contribution in bits.

For a registration archive, plausible values look like this. Forename agrees among true matches 88 times in 100, and among random pairs about 12 times in 1,000, so agreement is worth log base two of 0.88 divided by 0.012, which is log base two of 73.3, or 6.20 bits. Surname agrees among true matches 92 times in 100 and among random pairs 21 times in 10,000, giving log base two of 438, or 8.78 bits. Outward postcode agrees among true matches 71 times in 100 and among random pairs 43 times in 10,000, giving 7.37 bits.

Email domain is the interesting one. Among true matches it agrees only 45 times in 100, because people change employers and switch to webmail. Among random pairs it agrees 31 times in 1,000, because a large share of any file sits on a handful of consumer domains. Agreement is therefore worth log base two of 14.5, or 3.86 bits, and disagreement costs log base two of 0.55 divided by 0.969, which is minus 0.82 bits.

Sum the four for our pair: 6.20 plus 8.78 minus 0.82 plus 7.37 gives 21.53 bits.

Why does ranking beat a rule here?

Because the same evidence pattern means different things in different files, and a ranking lets you see that before you commit.

Run a second pair through the same arithmetic. Forename disagrees, Robert against Bob. Surname agrees. Email domain agrees, both on a large webmail provider. Postcode disagrees. That gives minus 3.04 plus 8.78 plus 3.86 minus 1.78, which is 7.81 bits. Both pairs would satisfy a loose rule requiring surname agreement plus one other field. The scores differ by 13.7 bits, a factor of about 13,000 in odds, and no rule can express that difference.

Ranking also tells you something a rule cannot: where the natural break in your file is. Sort every candidate pair by total weight and plot the counts. In a file with reasonable field coverage the distribution is bimodal, with a dense mass of low scoring pairs, a thin valley, and a second mass at the top. The valley is where the threshold belongs. If your distribution has no valley, that is worth knowing too, because it means your fields do not separate the two populations and no threshold will rescue the run.

What the prior does to the ranking

A weight of 21.53 bits sounds decisive on its own. It is not, until you say how rare matches are among the pairs you scored.

Suppose blocking left you with 4.6 million candidate pairs and you expect about 920 true matches among them, so roughly one pair in 5,000 is a match. The prior odds are 1 to 4,999, which in the same units is log base two of 1 divided by 4,999, or minus 12.29 bits. Add that to each score.

The first pair lands at 21.53 minus 12.29, which is 9.24 bits. That converts to odds of two to the power 9.24, about 600 to one, or a probability of 0.998. The second pair lands at 7.81 minus 12.29, which is minus 4.47 bits, odds of about 0.045 to one, a probability of 0.043. One is a merge and the other is not, and the difference came from the prior as much as from the fields.

This is where teams get burned. Loosen your blocking so that 46 million pairs are scored instead of 4.6 million, and the prior drops by another 3.3 bits, so every pair in the file becomes less likely to be a match without a single field changing. A score is meaningful only against the candidate set that produced it, which ties scoring to blocking key design more tightly than most pipeline diagrams admit.

Linking two files is a different job from deduplicating one

The acquisition case has a constraint that internal deduplication does not, and using it improves the result for free.

Your file holds 74,500 resolved people and the acquired file holds 38,000 rows, so the full comparison space is 74,500 times 38,000, about 2.83 billion pairs, which blocking cuts to the 4.6 million above. Within that space, each acquired row can correspond to at most one person in your file, and each of your people can absorb at most one acquired row. A row cannot be two people.

So after scoring, do not simply accept every pair above the threshold. Resolve conflicts: if one acquired row scores above the cut off against three of your people, take the highest and send the other two to review, because at least two of the three are wrong by construction. Teams that skip this step create merge chains, where A links to B, B links to C, and a cluster forms containing people who share nothing.

Deduplicating a single registration archive has no such constraint, since one person legitimately owns fifteen rows, and the pipeline there is the six stage one in duplicate attendee records across shows.

How much does this actually recover?

There is a published number worth carrying into a budget conversation. Enamorado, Fifield and Imai, writing in the American Political Science Review in 2019, merged two nationwide voter files of over 160 million records each using a probabilistic model. They matched about 95 per cent of voters, roughly 30 percentage points more than the exact matching method they compared against. At a 0.95 acceptance threshold their overall match rate was 93.04 per cent, with an estimated false discovery rate of 0.03 per cent and a false negative rate of 3.86 per cent.

The detail that should interest an organiser most is what happened to the movers. For people who had moved between states, the harder population to link, the probabilistic model found 20 times as many as the existing matching method. Your equivalent population is the buyer who changed employer between editions, and it is the population your exhibitors care about, because those are the people with new budgets.

Read those figures as a demonstration of the method rather than as a forecast for your file. Voter registration data carries a date of birth and a residential address, both of which are more discriminating than anything on a trade show badge. What transfers is the shape of the result: the gap between exact matching and scoring is largest exactly where the records changed, which is where your repeat attendance question lives.

Where this stops

The method assumes the two files describe overlapping populations of individuals, and it degrades badly when one of them is really a list of companies with a contact name attached. Exhibitor contact lists are usually the second kind. Matching those is a different problem with different fields and a different unit of resolution.

The second limit is missing data. If the acquired file has no mobile numbers, mobile cannot contribute a weight in either direction, so pairs from that file score systematically lower than pairs from yours. The remedy is to treat missing as its own comparison level with its own pair of probabilities, rather than as disagreement, and to report match rates alongside field completeness so nobody reads a coverage gap as a quality difference. The wider portfolio version of that argument is in cross event attendee matching.

The third is that a score is not a decision. Turning the ranking into merges needs two cut offs and an owner, which is the subject of the Fellegi Sunter model, and the estimates behind each field's two probabilities need their own work, covered in match weight calculation.

This week, take the acquired file and one of your own, block on the first three characters of the surname, and count the candidate pairs. Then count how many of those pairs agree on nothing but surname. That second number, divided by the first, tells you how much work the other fields have to do before any of this reaches your unified data layer, and it takes an hour.