Identity resolution.
One record per person across editions. Blocking, similarity measures, deterministic and probabilistic matching, survivorship, golden records, merges and unmerges.
20 articles · Unified data
- Cross event attendee matching, and why one buyer becomes four recordscross event attendee matchingCross event attendee matching explained: why exact email matching undercounts repeat buyers, and how the Fellegi-Sunter model weighs evidence instead.
- Duplicate attendee records across shows and the six stage fix for themduplicate attendee records across showsDuplicate attendee records across shows survive every dedupe you have run. Trace one buyer through ingest, blocking, scoring, decision, survivorship and golden record.
- One person many email addresses and why exact match joins lose herone person many email addressesWhen one person has many email addresses, an exact match join links none of them. Here is what agreement counting across surname, mobile and employer recovers instead.
- Deterministic matching rules and the exact point where they stop workingdeterministic matching rulesDeterministic matching rules fire on equality after normalisation. Measure a three tier cascade on 100 labelled pairs and you can see the tier where precision breaks.
- Probabilistic record linkage when your registration files share no common keyprobabilistic record linkageProbabilistic record linkage scores partial agreement across several weak fields when no shared key exists, then ranks every candidate pair by its total weight in bits.
- Fellegi Sunter model, applied to a five year registration archivefellegi sunter modelThe Fellegi Sunter model splits candidate pairs into link, non link and a middle region for review. Worked on two registration records across four fields, with both cut offs.
- Match weight calculation from m and u probabilities on badge scan datamatch weight calculationMatch weight calculation turns m and u probabilities into bits of evidence per field. Worked on badge scan data, where an email agreement is worth 9.8 bits and a surname less.
- The conditional independence assumption and where event data quietly violates itconditional independence assumptionThe conditional independence assumption lets match weights be added across fields. Employer name and work email domain break it, and the double counting is worth four bits.
- Blocking for entity resolution before the pair count runs to billionsblocking for entity resolutionBlocking for entity resolution cuts 88 billion candidate pairs to 50 million, and ten oversized blocks produce most of what is left. Here is the arithmetic and the cost.
- Blocking key design that protects recall without exploding the comparison countblocking key designBlocking key design is a trade between pair completeness and reduction ratio. Measure both on 500 labelled pairs, then price each extra pass by the comparisons it costs.
- The sorted neighbourhood method for matching exhibitor contact lists year to yearsorted neighbourhood methodThe sorted neighbourhood method sorts records on a composite key and compares inside a sliding window, cutting 1.25 billion exhibitor pairs to 450,000 in one pass.
- Candidate pair generation and the recall you give away before scoring startscandidate pair generationCandidate pair generation sets a hard ceiling on match recall, because any true pair it discards can never be scored. How to measure that ceiling and raise it.
- Jaro Winkler similarity and why it favours the first four charactersjaro winkler similarityJaro Winkler similarity explained by hand on two badge names, including the prefix bonus, the transposition rule, and the fields where the bonus misleads you.
- Jaccard token overlap for company names on an exhibitor directoryjaccard token overlapJaccard token overlap scores company names by shared words instead of shared characters, which is why it handles a missing word on an exhibitor directory better.
- Levenshtein distance for names and where it misleads on short surnameslevenshtein distance for namesLevenshtein distance for names counts insertions, deletions and substitutions, and a fixed threshold of 2 merges half the four letter surnames in your file.
- Phonetic name matching with Soundex and Metaphone on international attendee listsphonetic name matchingPhonetic name matching with Soundex and Metaphone recovers variant spellings on an international attendee list, and the false pairs it creates are countable.
- Choosing a string similarity measure for each field on a registration recordchoosing a string similarity measureChoosing a string similarity measure works field by field: the failure mode of a forename differs from a company name, so one function everywhere loses on both.
- Company name normalisation before you compare a single exhibitor recordcompany name normalisationCompany name normalisation is its own pipeline stage: case, punctuation, accents and ampersands are resolved before any similarity measure gets to see the name.
- Legal entity suffixes and the exhibitor records they split in twolegal entity suffixesLegal entity suffixes split one exhibitor into two accounts. Strip them before comparison, then keep the stripped form as a jurisdiction signal a reviewer can use.
- Parent and subsidiary matching when one exhibitor books under four legal namesparent and subsidiary matchingParent and subsidiary matching keeps four contracting entities separate while rolling booth spend up to one group, using an edge instead of a merge.