Phonetic name matching with Soundex and Metaphone on international attendee lists

Look at the surname column of a registration file for a show with a genuinely international audience. Polish surnames typed by a Polish delegate and by a British sales assistant. Arabic names transliterated four ways by four registration routes. Turkish names with the dotless i either preserved or flattened. The same person, in two editions, under two spellings that no exact join will ever bring together.

Phonetic name matching is the oldest fix for this and it still earns its place, mostly as a blocking key rather than as a score. The gain is real and the cost is real, and both can be counted on your own file before you commit to it.

What Soundex actually does

Soundex reduces a surname to a letter and three digits. The coding rules published by the US National Archives, which maintains the version used for the census indexes, assign 1 to B, F, P and V, 2 to C, G, J, K, Q, S, X and Z, 3 to D and T, 4 to L, 5 to M and N, and 6 to R. Vowels along with H, W and Y are disregarded. The first letter of the surname is kept as written, doubled letters count once, and different letters sitting side by side with the same digit also count once.

Work Smith. Keep the S. The M is 5, the I is disregarded, the T is 3, the H is disregarded. That gives S-53, padded to S-530.

Work Smyth. Keep the S. The M is 5, the Y is disregarded, the T is 3, the H is disregarded. S-530 again.

Two spellings, one key, and the pair becomes reachable by a matcher that would otherwise never compare them. That is the entire proposition.

What does phonetic encoding buy you on an international list?

More than you would guess from the English examples, because transliteration differences are mostly vowel differences and Soundex throws vowels away.

Mohammed keeps the M, the H is disregarded, the second M is 5, the doubled M counts once, and the D is 3. M-530. Muhammad reduces to M-530. Mohamed reduces to M-530. Three transliteration conventions, three registration routes, one key.

Kowalski keeps the K, the W is disregarded, the L is 4, then S and K sit side by side with the same digit 2 and count once, and the trailing I is disregarded. K-420. Kowalsky gives K-420 as well.

Yilmaz keeps the Y, the L is 4, the M is 5, the Z is 2. Y-452. Yilmez gives Y-452.

Put a number on it. Take 500 pairs you know to be the same person across two editions. Under exact surname blocking, suppose 431 of them have identical surname strings, so 69 pairs are unreachable before any scoring happens. Under Soundex blocking, suppose 468 survive. You have recovered 37 pairs, which moves the reachable share from 86.2 per cent to 93.6 per cent.

What it costs

The same insensitivity that collapses three spellings of Mohammed collapses names that have nothing to do with each other.

Nowak keeps the N, the W is disregarded, the K is 2. N-200. Nowicki keeps the N, the W is disregarded, then C and K sit side by side with the same digit and count once. N-200. Two of the most common Polish surnames, one key.

Robert reduces to R-163 and so does Rupert. Chen, Chin, Chun and Chan all reduce to C-500, since the vowels are the only thing separating them.

Continue the arithmetic. Suppose exact surname blocking produced 240,000 candidate pairs, of which 431 are true, so one candidate pair in 557 is worth merging. Suppose Soundex blocking produces 1,050,000 candidate pairs, of which 468 are true, so one pair in 2,244 is worth merging. The candidate set is now four times more dilute for a gain of 7.4 points of reachable recall.

Whether that trade is good depends on what happens downstream. If the scoring stage is strong, dilution costs compute and little else, because a weak pair scores low and never reaches a person. If your pipeline sends everything above a modest threshold to a review queue, dilution costs staff hours directly, and 810,000 extra pairs is not a rounding error. The rule that follows is worth stating plainly: use phonetic codes to widen candidate pair generation, and never use a phonetic code agreement as evidence in the score itself. It is far too coarse to weigh.

Why does the first letter break transliterated names?

Because Soundex keeps it verbatim, and on an international list the first letter is the least stable character in the field.

Hussein encodes as H-250. The same person registering as Al Hussein encodes as A-425, since the A is now the retained letter and the L becomes the first digit. Nothing in the code survives the change. Van Der Berg encodes as V-536 while Berg alone encodes as B-620.

This is the failure that matters most for an exhibition with Gulf, Dutch, Belgian or Iberian exhibitors, because the particle is stored inconsistently by every system you own. A field the delegate typed once as Al Rashid appears in the badge file as Rashid and in the CRM as Al-Rashid, and the phonetic key differs across all three.

The fix is a normalisation rule ahead of the encoder. Swapping in a stronger encoder does nothing here, because every one of them will read Al Rashid and Rashid as different first sounds. Strip a known list of particles into a separate field, encode the remainder, and keep the particle for display and for the golden record. Doing it in that order means the code is computed on the part of the name that stays constant. Any string measure sitting downstream, including the prefix weighted comparator in J12, benefits from the same reordering, since it also reads the first characters as the most informative.

Metaphone, and the second code it returns

Lawrence Philips published Metaphone in Computer Language in December 1990 as a replacement for Soundex that encodes the sound of letter groups instead of assigning one digit per consonant. It knows that PH sounds like F, that a leading K in KN is silent, that CH behaves differently in different positions, and it produces a variable length consonant code rather than a fixed three digits.

Philips followed it with Double Metaphone in the C/C++ Users Journal in 2000, and the addition in the name is the useful part. A name with two plausible pronunciations receives a primary code and a secondary code, and two names are treated as a phonetic match when any of their codes agree. That handles the case where the same spelling is pronounced one way by an English speaker and another by a Polish or German speaker, which on an exhibition floor is most of the time.

Christen surveyed the phonetic encodings, Soundex and Phonex and Phonix and NYSIIS and Double Metaphone among them, and compared twenty name matching functions across four datasets of personal names, at the ICDM workshops in 2006. He describes Double Metaphone as the encoding that aims to account better for names outside English. That is the reason to prefer it as a blocking key on a portfolio with genuinely international audiences, with one caveat: the codes are longer and more specific, so the blocks are smaller and the dilution is lower, and the recall gain is correspondingly smaller too. Measure both on your 500 labelled pairs before choosing.

Where this stops

Phonetic encoding models one language's sound system and then gets applied to all of them. Metaphone encodes English pronunciation, and a Turkish delegate's surname is not being pronounced in English by anyone who matters.

The consequence is uneven accuracy across your audience, in a direction that is easy to miss. Your match rate will look better for delegates with names the encoder was designed around and worse for everybody else, so a report of match rate by country will show the international segments performing badly, and somebody will conclude that international registration data is lower quality. The data is the same quality. The method knows less about it.

The second limit is that no phonetic code survives a script change. A name recorded in Arabic script in one system and in Latin transliteration in another produces no shared key at all, because the encoder needs Latin characters to work with. Transliteration has to happen first, it is lossy, and different transliteration standards give different results for the same name.

The third is that phonetic agreement is weak evidence and looks like strong evidence, because a matching code feels like a match. Two records agreeing on S-530 have told you very little, since S-530 covers Smith, Smyth, Smithe, Smithey and a long tail of others. Treat it as a reason to compare, never as a reason to merge, and let a length aware string measure such as the normalised edit distance in J14 do the work of separating the pairs the code brought together.

This week, encode the surname column of one show's registration file with Soundex, count the distinct codes, and sort the code groups by size. The largest ten groups tell you immediately where your false pairs will come from, and if one of them contains four unrelated common surnames from a single national market, you have found the block that will flood the review queue in the unified data pipeline before you have written a line of matching logic.