Identity resolution.
One record per person across editions. Blocking, similarity measures, deterministic and probabilistic matching, survivorship, golden records, merges and unmerges.
30 articles · Unified data
- Cross event attendee matching, and why one buyer becomes four recordscross event attendee matchingCross event attendee matching explained: why exact email matching undercounts repeat buyers, and how the Fellegi-Sunter model weighs evidence instead.
- Duplicate attendee records across shows and the six stage fix for themduplicate attendee records across showsDuplicate attendee records across shows survive every dedupe you have run. Trace one buyer through ingest, blocking, scoring, decision, survivorship and golden record.
- One person many email addresses and why exact match joins lose herone person many email addressesWhen one person has many email addresses, an exact match join links none of them. Here is what agreement counting across surname, mobile and employer recovers instead.
- Deterministic matching rules and the exact point where they stop workingdeterministic matching rulesDeterministic matching rules fire on equality after normalisation. Measure a three tier cascade on 100 labelled pairs and you can see the tier where precision breaks.
- Probabilistic record linkage when your registration files share no common keyprobabilistic record linkageProbabilistic record linkage scores partial agreement across several weak fields when no shared key exists, then ranks every candidate pair by its total weight in bits.
- Fellegi Sunter model, applied to a five year registration archivefellegi sunter modelThe Fellegi Sunter model splits candidate pairs into link, non link and a middle region for review. Worked on two registration records across four fields, with both cut offs.
- Match weight calculation from m and u probabilities on badge scan datamatch weight calculationMatch weight calculation turns m and u probabilities into bits of evidence per field. Worked on badge scan data, where an email agreement is worth 9.8 bits and a surname less.
- The conditional independence assumption and where event data quietly violates itconditional independence assumptionThe conditional independence assumption lets match weights be added across fields. Employer name and work email domain break it, and the double counting is worth four bits.
- Blocking for entity resolution before the pair count runs to billionsblocking for entity resolutionBlocking for entity resolution cuts 88 billion candidate pairs to 50 million, and ten oversized blocks produce most of what is left. Here is the arithmetic and the cost.
- Blocking key design that protects recall without exploding the comparison countblocking key designBlocking key design is a trade between pair completeness and reduction ratio. Measure both on 500 labelled pairs, then price each extra pass by the comparisons it costs.
- The sorted neighbourhood method for matching exhibitor contact lists year to yearsorted neighbourhood methodThe sorted neighbourhood method sorts records on a composite key and compares inside a sliding window, cutting 1.25 billion exhibitor pairs to 450,000 in one pass.
- Candidate pair generation and the recall you give away before scoring startscandidate pair generationCandidate pair generation sets a hard ceiling on match recall, because any true pair it discards can never be scored. How to measure that ceiling and raise it.
- Jaro Winkler similarity and why it favours the first four charactersjaro winkler similarityJaro Winkler similarity explained by hand on two badge names, including the prefix bonus, the transposition rule, and the fields where the bonus misleads you.
- Jaccard token overlap for company names on an exhibitor directoryjaccard token overlapJaccard token overlap scores company names by shared words instead of shared characters, which is why it handles a missing word on an exhibitor directory better.
- Levenshtein distance for names and where it misleads on short surnameslevenshtein distance for namesLevenshtein distance for names counts insertions, deletions and substitutions, and a fixed threshold of 2 merges half the four letter surnames in your file.
- Phonetic name matching with Soundex and Metaphone on international attendee listsphonetic name matchingPhonetic name matching with Soundex and Metaphone recovers variant spellings on an international attendee list, and the false pairs it creates are countable.
- Choosing a string similarity measure for each field on a registration recordchoosing a string similarity measureChoosing a string similarity measure works field by field: the failure mode of a forename differs from a company name, so one function everywhere loses on both.
- Company name normalisation before you compare a single exhibitor recordcompany name normalisationCompany name normalisation is its own pipeline stage: case, punctuation, accents and ampersands are resolved before any similarity measure gets to see the name.
- Legal entity suffixes and the exhibitor records they split in twolegal entity suffixesLegal entity suffixes split one exhibitor into two accounts. Strip them before comparison, then keep the stripped form as a jurisdiction signal a reviewer can use.
- Parent and subsidiary matching when one exhibitor books under four legal namesparent and subsidiary matchingParent and subsidiary matching keeps four contracting entities separate while rolling booth spend up to one group, using an edge instead of a merge.
- Email address normalisation and the duplicate registrations it removes for freeemail address normalisationEmail address normalisation lowercases the domain, then applies provider specific rules for dots and plus tags, collapsing four registration rows onto one match key.
- Phone number normalisation to a single international format before matchingphone number normalisationPhone number normalisation parses every registration row to E.164 with an inferred country code, so four spellings of one mobile collapse to a single match key.
- Address matching for exhibitor and attendee records across five years of filesaddress matchingAddress matching across five years of exhibitor files has to work on parsed components, because whole string comparison rewards the shared city tail.
- Match threshold selection and what each point of score actually costs youmatch threshold selectionMatch threshold selection starts from a written error budget, then reads the cut off off a labelled sample. Each point of score has a price in false merges.
- The clerical review band and how wide you can afford to make itclerical review bandThe clerical review band between two match cut offs converts into a decision count, so widening it by five points of score is a staffing commitment.
- Resolving contacts to companies when the badge says a domain you never sawresolving contacts to companiesResolving contacts to companies by email domain covers a registration file quietly and wrongly, because consultants carry a domain they do not buy for.
- Survivorship rules that decide which value wins on a merged attendee recordsurvivorship rulesSurvivorship rules choose a winning value per attribute, so a merged attendee record can take its name from one source row and its phone number from another.
- Golden record construction after the merge decision has already been madegolden record constructionGolden record construction assembles one record from surviving values and keeps provenance per field, so every attribute traces to a source row.
- The most recent wins rule and the two fields it gets wrongmost recent wins ruleThe most recent wins rule is right for job title and phone and wrong for name and email, where the newest row is often a hurried kiosk registration.
- Conflicting values across sources and how to pick one without guessingconflicting values across sourcesConflicting values across sources are settled by ranking each source per field, because a system accurate on company is often poor on job title.