The conditional independence assumption and where event data quietly violates it

A team adds a field to their matcher. Work email domain, extracted from the address they already hold, straightforward to compute, obviously informative. Merge counts rise about 14 per cent. Everyone is pleased for a fortnight, until the complaints arrive and a sample shows the extra merges are worse than the ones the model was already making.

The conditional independence assumption is why. Sum the match weights across fields, as every implementation of the classic model does, and you are asserting that once you know whether a pair is a true match, agreement on one field tells you nothing about agreement on another. Employer name and work email domain are the same fact written twice, so the model counted one piece of evidence as two.

The assumption, stated plainly

Winkler, writing for the US Census Bureau in 2006, sets out the standard model and notes that the conditional independence assumption corresponds exactly to the naive Bayes assumption in machine learning. It is what licenses the addition. The probability of the whole agreement pattern given a match is taken to be the product of the individual field probabilities, so the logarithm of the ratio becomes a sum of per field weights, and a matcher becomes something you can compute in one pass and explain in a meeting.

The assumption is never exactly true and it does not need to be. It needs to be approximately true for the fields you actually use, and the practical question is which pairs of your fields break it badly enough to matter.

Where does event data break it?

Four places, and the first of them is nearly universal in registration data.

• Employer name and work email domain. The domain was derived from the employer in most registration systems, or the employer was derived from the domain by an enrichment step. Either way, if one agrees the other almost certainly does.
• City and postcode. Agreement on a full postcode implies agreement on the city. The second field adds almost nothing once the first has fired.
• Job title and seniority band. If your pipeline computed the band from the title, the band contains no information the title did not already carry, and adding it as a separate comparison is double counting by construction.
• Show and industry sector. Two people registering for a plastics show are both in plastics, so sector agreement among random pairs within one show is close to certain, which makes the field nearly worthless there while looking valuable in a portfolio wide run.

Winkler gives the same structure with a cleaner example: agreement on a surname and agreement on the Soundex code of that surname carry identical information, so the weight for the second should be set to zero. Any pair of fields where one is a function of the other has this property, and enrichment pipelines manufacture such pairs constantly.

The arithmetic of double counting

Work it on the employer pair, using values a mid sized registration archive would produce.

Employer name agreement: m of 0.72, u of 0.019, giving log base two of 37.9, which is 5.24 bits. Work email domain agreement: m of 0.66, u of 0.031, giving log base two of 21.3, which is 4.41 bits. Added naively, a pair agreeing on both collects 9.66 bits.

Now measure the two fields together instead of separately, which takes one extra query against your labelled pairs. Among true matches, both agree 63 times in 100, so the joint m is 0.63. Among non matches, both agree 14 times in 1,000, so the joint u is 0.014. The correct weight for the combined comparison is log base two of 0.63 divided by 0.014, which is log base two of 45, or 5.49 bits.

The naive sum overstates the evidence by 9.66 minus 5.49, which is 4.17 bits. Four bits is a factor of about 18 in the odds. A pair that the model reports at 99 to one is really at about 5.5 to one, and if your upper cut off sits anywhere near that region, you are auto merging pairs a person would have rejected.

The conditional probabilities show what happened. Among true matches, the domain agrees in 0.63 of 0.72, which is 87.5 per cent of the cases where the employer already agreed. Among non matches it is 0.014 of 0.019, or 74 per cent. Both are far above the unconditional rates, which is what dependence looks like when you go and measure it.

Take that back to the team from the opening. Their model made 22,400 automatic merges before the new field and 25,500 after, so 3,100 pairs crossed the upper cut off purely because a correlated field pushed them over. A reviewer judged 60 of those 3,100 and found 11 wrong, a precision of 0.82 against the 0.98 the model had been holding on the pairs it merged before. The extra 3,100 merges were bought at roughly 570 bad merges across the file, and nobody would have signed that off if it had been written down that way.

How do you fix it without building a new model?

The cheapest fix is to stop treating the two fields as two fields.

Replace employer name and work email domain with a single comparison having three or four levels: both agree, employer agrees and domain differs, domain agrees and employer differs, neither agrees. Estimate m and u for each level directly from labelled pairs. The additivity of the rest of the model survives, the double counting disappears, and the interesting middle levels now carry real information, since employer agreeing while the domain differs is a genuine signal that somebody typed the company name rather than inheriting it from an address.

The second option is to drop one of the pair. If email domain adds 4.41 bits of which most is already in the employer field, deleting it costs you little and removes the failure mode entirely. I would drop rather than combine when the two fields agree with each other more than 90 per cent of the time among true matches, because at that point the second field is a copy.

Whichever of the two you choose, check the result the same way. Re run the model on the labelled sample, compare the score each pair received before and after, and look at the pairs that crossed the upper cut off in either direction. If the combined comparison is working, the pairs that fall below the cut off should be the ones where the only two agreements were the correlated pair, and a reviewer should agree that they were never safe to merge.

The third option is a model that represents the dependence explicitly. Larsen and Rubin, writing in the Journal of the American Statistical Association in 2001, fit mixture models with an expectation maximisation algorithm that can represent dependence between field agreements instead of assuming it away, and Winkler's 2006 overview names their work as one of the established ways of dealing with lack of independence. This is the right answer for a statistics agency and a heavy lift for an event data team, so treat it as the destination and the combined comparison as the road.

The field ceiling nobody mentions

There is a line in Winkler's overview that should be printed above every matching backlog: having more than six to ten fields for matching is generally not needed.

That runs against the instinct of every team that has ever tried to improve a matcher, which is to add another field. The reason more fields stop helping is exactly the dependence above. Registration records do not contain twenty independent facts about a person. They contain a name, a way to contact them, a place, an employer, and a set of attributes derived from those four. Once you have priced the underlying facts, the additional columns are mostly restatements, and each restatement quietly inflates the totals for pairs that agree on the fact behind them.

So the productive direction after six good fields is not a seventh. It is better comparison levels on the fields you have, better normalisation before comparison, and value specific weights so a rare employer counts for more than a common one.

Where this stops

Measuring dependence needs labelled pairs, and the joint probabilities need more of them than the marginal ones do. Estimating m for employer alone from 200 labelled matches is reasonable. Estimating the joint probability for a four level combined comparison from the same 200 leaves you with cells containing a handful of pairs each, and the weights inherit that noise. Either label more pairs or use fewer levels.

The second limit is that dependence is not stable across your file. Employer and domain are tightly coupled for corporate registrants and almost uncoupled for the self employed and for anyone using webmail, so a single combined comparison is an average over two populations that behave differently. Splitting the estimate by whether the address is corporate or free webmail handles most of it, at the cost of another parameter set to maintain.

The first step takes one query. On your labelled pairs, or on pairs that agree exactly on email and mobile if you have no labels yet, count how often the work email domain agrees given that the employer name agrees. If that number is above 0.9, your matcher is double counting today, and the weights in match weight calculation are overstated by however many bits the second field is contributing. Fix the comparison before you touch the cut offs in the Fellegi Sunter model, because moving a threshold to compensate for an inflated score corrupts every other pair in the unified data layer.