False merge rate is the number your stewards should watch most closely

A sales director calls about an account. The record says the buyer attended in 2024 and sat in two conference sessions, and the buyer has told him, twice, that she has never been to the show. He wants to know which system is lying.

Neither is. Two records were merged eighteen months ago because both carried the same surname, the same dealership and the same postcode, and they belong to a father and a daughter who both work there. The merge produced no error, no flag and no queue entry. It just quietly became true. False merge rate is the measurement that would have told you how often that happens, and almost nobody running an identity pipeline has the number.

Why a bad merge costs more than a missed one

The two errors are not symmetric and should never be traded against each other as though they were.

A missed match leaves you with two records for one person. Somebody gets two emails, the audience count is slightly high, and the day it starts to matter you can find the pair and join them. The evidence survives.

A false merge takes two people and produces one. The surviving record carries a mixture of both histories, and the losing record's attributes are gone from every view your team uses. One of those two people has now dropped out of your marketable audience without appearing anywhere as a loss. If your survivorship rules picked the more recent email, the person who stops receiving your campaigns is the one who moved jobs most recently, which is disproportionately the senior buyer you most wanted.

The failure also propagates. Once the merged identifier has been sent to a CRM, an email platform and an exhibitor portal, unpicking it means unpicking three systems that have since written their own rows against it.

Wang and Strong's 1996 framework in the Journal of Management Information Systems puts believability inside intrinsic data quality, alongside accuracy, objectivity and reputation, on the finding that data consumers judge quality on more than correctness. A file full of duplicates looks untidy and stays believable. A file where two known people have become one is the kind of defect that ends the sales team's willingness to use the system at all, which is a bigger loss than the record.

How do you measure a false merge rate?

Sample the merges you already made, and have a person label them.

Pull a random sample from the pairs the system merged automatically in one edition. Random matters: a steward reviewing whichever merges look odd will find a high rate, and the number will be unusable. For each pair, the steward sees both original records as they were before survivorship ran, and answers one question. Same person or different people. Not sure counts as its own answer and gets reported separately, because burying uncertain cases in either bucket biases the result.

The rate is false merges over pairs reviewed. That is the whole calculation, and its usefulness comes from doing it every edition with the same instructions.

Black and van Nederpelt's 2020 paper for the DAMA NL Foundation sets out the three ways duplication can go wrong, and their third case is the one being measured here: one record having the same key as another when the two are different things, which they call a false duplicate. Naming it helps, because teams routinely report a single accuracy figure for a matching system and mean only the first two cases.

The arithmetic on 200 labelled pairs

Say the pipeline auto merged 4,200 pairs for one edition. The steward reviews 200 of them and marks 8 as different people.

The point estimate is 8 divided by 200, which is 4 per cent. The standard error on a proportion is the square root of p times one minus p over n, so the square root of 0.04 times 0.96 divided by 200, which is the square root of 0.000192, or 0.0139. Roughly 1.4 points.

A 95 per cent interval is the estimate plus or minus 1.96 standard errors, so 4 per cent plus or minus 2.7 points, giving a range of about 1.3 to 6.7 per cent.

Apply that to the 4,200 merges and the honest statement is that somewhere between 54 and 282 pairs of people in this edition were wrongly collapsed into one. That range is wide, and it is the truth. Reporting 168 as though it were measured would be the same defect the exercise exists to catch.

Doubling the sample to 400 pairs takes the standard error to the square root of 0.04 times 0.96 divided by 400, which is 0.0098, and the interval to roughly 2.1 to 5.9 per cent. Four hundred labels for 0.8 of a percentage point on each side. Whether that is worth a steward's morning depends on what decision is waiting on it, and usually it is not.

What the number tells you to change

A false merge rate is a reading on the auto merge threshold, and it is the only reading that speaks to the expensive direction of error.

If the rate comes back at 4 per cent and the pairs a steward marked as wrong cluster around scores just above the auto merge line, raise the line. The pairs that stop merging automatically do not disappear. They land in the review queue, and 700 pairs moving out of auto merge is 700 more decisions a person has to make, on a queue that already has a length somebody is managing. That trade is the real content of a threshold decision, and it is a queueing problem with its own arithmetic, which is K30.

If the rate comes back near zero, your threshold is probably too high and you are leaving joins on the table, which will show up as a stubborn headline duplicate rate in K24 that no amount of tuning shifts.

Two more things belong in the same report. The share of labelled pairs the steward could not decide, because a high not sure share means your review interface is not showing enough context. And the field pattern behind the false merges, since eight bad merges that all agreed on surname, company and postcode tell you that combination is doing too much work in the score.

What happens when two stewards disagree?

The measurement instrument is a person, so it needs its own check.

Give the same 50 pairs to two stewards without telling either. Count how often they agree. If they agree on 46, you have 92 per cent agreement and a number you can broadly trust. If they agree on 34, your false merge rate has an error bar wider than the thing you are trying to measure, and the fix is the instructions rather than the sample size.

Disagreement usually traces to one of two cases. Shared identity, where two people at a small firm share a surname and an address, and the evidence genuinely points both ways. And stale evidence, where the same person appears with two employers and one steward treats the employer as identifying and the other does not. Both are worth writing down as rules, with an example each, before the next round of labelling.

Where the measurement stops

Sampling the merges you made tells you nothing about the pairs the system never considered.

If your blocking step never put two records in the same block, they were never scored, never merged and can never appear in your sample. The false merge rate can be excellent while the pipeline is quietly failing to find half the true matches, and those two facts are measured on different populations by different exercises.

The rate also degrades as a guide the moment anything upstream changes. A new registration platform, a change to which fields are mandatory on the form, or a source that starts supplying company names in a different format will all move the score distribution, and a false merge rate measured before the change is describing a system that no longer exists. Re-measure each edition, keep the previous figure, and report the pair.

The last limit is that a steward labelling a pair from two registration records is making a judgement with the same information the model had. Where the truth is genuinely unavailable, the label is a considered guess, and the honest version of the metric reports the not sure share next to it instead of forcing every pair into a verdict.

Take 200 auto merged pairs from your most recent edition this week, put both original records side by side, and have one person label them. The rate you get is probably the first defensible number anyone in your organisation has had about the cost of automatic matching, and it belongs in the same unified data quality report as the duplicate figure it trades against. If you have source tags on the records, keep them attached to the sample, because the false merges usually cluster in one feed and attributing them back to their origin is K26's job.