Deterministic matching rules and the exact point where they stop working

The matching specification for one organiser's registration warehouse ran to eleven deterministic matching rules, written over three years by four people, applied in an order nobody could explain. Two of them had been added after a complaint. One had been commented out and left in place. Nobody knew which rule was responsible for any given merge, and nobody could say what would break if a rule were removed.

That is the failure mode for rules, and it has nothing to do with the theory. Rules are fine. They are fast, they run in SQL, and a person can read one and say whether it is reasonable. The trouble starts when a cascade grows without measurement, because each new tier is added to catch a case somebody complained about and no one checks what else it caught.

Doidge and Harron, writing in the International Journal of Population Data Science in 2018, put the distinction plainly: deterministic linkage is based on rules and probabilistic linkage is based on weights or scores, and each pattern of agreement across the matching fields corresponds to a potential decision rule in the first and a match weight in the second. They also describe how such rules are used in practice, implemented sequentially and usually starting with those thought least likely to return false matches.

The three tier cascade, written out

Here is a cascade for a registration archive, strictest first. Every field is normalised before comparison: lowercased and plus tag stripped for email, parsed to international format for mobile, legal suffixes and punctuation removed for employer, whitespace removed and uppercased for postcode.

• Tier 1. Email equal.
• Tier 2. Surname equal and mobile equal.
• Tier 3. Surname equal and outward postcode equal and normalised employer equal.

The ordering is the design. A pair that satisfies tier 1 never reaches tier 2, so each tier only ever sees the pairs that everything above it failed to link. That property is what makes the cascade measurable, and it is the thing most implementations throw away by running all the rules and unioning the results.

What 100 labelled pairs tell you about each tier

Take 100 candidate pairs from your own file, judged by a person who knows the sector, with the answer recorded as same person or different people. This is an afternoon of work and it is the only way to know what your rules do.

Draw the sample carefully, because the obvious method wastes the afternoon. A pair picked uniformly at random from 100,000 registration rows is drawn from about 5 billion possible pairs, of which a few hundred thousand at most are true matches, so a uniform sample of 100 pairs will contain no true matches at all and tell you nothing. Sample instead from pairs that share a block, and stratify: 40 pairs that some tier links, 40 that no tier links but that agree on at least one field, and 20 drawn from the rest. Record the strata so you can weight the results back to the file.

Suppose the 100 pairs contain 74 true matches. Run the cascade and record which tier fired.

Tier 1 links 48 pairs. All 48 are true. Precision is 48 divided by 48, which is 1.00, and it recovers 48 of the 74 true pairs, so recall so far is 0.65.

Tier 2 links 17 more. Sixteen are true, one is a father and son at the same firm sharing a landline entered as a mobile. Precision for this tier is 16 divided by 17, or 0.94. Cumulative recall is 64 of 74, which is 0.86.

Tier 3 links 14 more. Nine are true and five are different people at the same employer in the same postcode district with a shared surname. Precision for the tier is 9 divided by 14, or 0.64. Cumulative recall reaches 73 of 74, or 0.99, and cumulative precision falls to 73 divided by 79, which is 0.92.

Now the trade is visible. Tier 3 buys you 9 true pairs and costs you 5 false merges. Whether that is a good deal depends on what a false merge costs in your business, and in an event business it costs more than a duplicate, because a wrongly merged buyer stops receiving invitations and vanishes from the audience count without anybody noticing. I would not ship tier 3 as written. I would tighten it by requiring the forename to agree as well, accept a lower recall, and send the pairs it no longer links to a review queue.

Where is the exact point they stop working?

The point is not a data volume and it is not a field count. It arrives when the marginal tier's precision falls below the level your merge policy can absorb, and that happens for a reason you can see in the arithmetic above.

Every tier is a conjunction of equalities. Adding a field to a tier raises its precision and lowers its recall. Removing a field does the reverse. What you cannot do with a rule is say that agreement on the surname Zbigniew is worth more than agreement on the surname Smith, so a tier tuned to be safe for common surnames is needlessly strict for rare ones, and a tier tuned for rare surnames merges the wrong Smiths. The cascade has one dial and the file needs two.

The second failure is silent and worse. A rule can only fire on equality, so any typing error, any nickname, any transposed pair of letters removes the pair from every tier that uses that field. Tier 1 misses every genuine repeat where the person used a different address, which is the ordinary case rather than the exception, as one person with many email addresses sets out.

When are deterministic rules good enough?

More often than the modelling literature implies, and there is a measured answer to point at. Zhu, Matsuyama, Ohashi and Setoguchi ran 96 simulated scenarios in the Journal of Biomedical Informatics in 2015, varying the discriminative power of the linking fields, the missing data rate, the error rate and the file size. Probabilistic linkage came out ahead across the scenarios, with a better trade off between sensitivity and positive predictive value. Deterministic linkage performed comparably where the data was exceptionally clean, with error rates under about five per cent, and it did so with substantially less computation.

Read that against your own file rather than against a benchmark. A single edition of a single show, registered through one platform in one season, often has field error rates in that range, and a cascade will do the job. Five years of history across eight shows, three registration platforms and two acquisitions will not, because the error rate you care about includes every difference introduced by a form change, a migration and a person changing employer.

My own position is that the cascade earns its place even when you are running a model. Run it first, in its own stage, and treat the pairs it links at tier 1 as settled. That gives you a fast path for most of the file, a set of high confidence pairs to estimate model parameters from, and a diagnostic that catches normalisation bugs early. What the cascade should not be is the whole matcher, and the case for what comes next is in probabilistic record linkage.

The match rank you should be storing

Doidge and Harron make a point that costs nothing to implement and gets skipped anyway. The step at which a link is identified can be used as an indicator of confidence about that link.

So store it. Every link row in your match table should carry the tier that produced it, the version of the rule set, and the date. Then a steward investigating a complaint can see that this particular merge came from tier 3 under rule set version 4, and an analyst can rebuild last year's repeat rate under this year's rules to find out how much of the change was people and how much was you. Without that column, a rule change is indistinguishable from an audience change, and both show up in the same report.

Storing the rank also lets you retire a tier without an argument. If tier 3 produced 4,100 links last year and 62 of the 80 sampled were wrong, that is a decision anybody can make in a meeting. If all you have is a merged person table, the same decision requires a rebuild.

Where this stops

The cascade cannot express partial agreement, and that limit is structural. Kathryn against Katherine is either equal or unequal to a rule, and it is neither to a person. You can approximate the difference by putting normalisation in front of the comparison, adding a nickname table, or comparing a phonetic code instead of the raw string, and each of those is a small model wearing a rule's clothing. That is worth knowing when someone claims a rule based system has no assumptions in it.

The second limit is that precision measured on 100 pairs has an interval around it, and a thin one. Nine true out of 14 is a point estimate with a confidence interval running roughly from 0.35 to 0.87. Treat a single small sample as a signal to investigate rather than as an accuracy figure, and label more pairs for any tier you are about to make automatic.

Start with an inventory. List every rule currently running against your registration data, in order, and for each one count how many links it produced last year. Most teams find that two tiers produce 90 per cent of the links and the rest produce arguments. Then label 100 pairs against the smallest tier and see whether it deserves to stay in the unified data layer at all, before you spend a quarter on the decision theory in the Fellegi Sunter model.