Jaccard token overlap for company names on an exhibitor directory

An exhibitor directory with 2,400 companies in it holds perhaps 300 firms twice. The usual cause is ordinary. A business books stand space through a contracting entity one year and a divisional name the next, and a salesperson types whatever was on the purchase order.

Jaccard token overlap is the measure that handles this class of difference well, and the reason is narrow enough to state in a sentence. The gap between two versions of a company name is almost always a whole word, so a measure that counts words beats a measure that counts characters.

Paul Jaccard published the coefficient in New Phytologist in 1912, comparing which plant species two alpine districts had in common. The arithmetic is the same on a company name. Count what both have, divide by everything either has.

How does token overlap score two company names?

Split each name into tokens on whitespace and punctuation, lowercase them, drop the legal suffix, and compare the resulting sets.

Take four rows out of the directory, all of them the same firm.

Brightwell Packaging Systems Limited becomes the token set brightwell, packaging, systems. Brightwell Packaging Systems is the same three tokens. Brightwell Packaging is two tokens. Brightwell Group is brightwell and group.

The first pair share three tokens and have three distinct tokens between them, so the score is 3 over 3, which is 1.0. Perfect agreement, achieved by stripping one word that carries no identifying information, a step that belongs to legal entity suffix handling in J18.

The second comparison is the interesting one. Brightwell Packaging Systems against Brightwell Packaging shares brightwell and packaging, and the union across both names is brightwell, packaging and systems. Two over three is 0.6667. Call it 0.67.

Brightwell Packaging against Brightwell Group shares only brightwell, against a union of three, so 1 over 3, which is 0.33.

Those three numbers behave the way a person would. Identical after suffix stripping, close with a word missing, weak when only the trading name survives.

Why does a token measure beat a character measure here?

Run the same pair through an edit based comparison and watch the number fall apart.

Brightwell Packaging Systems Limited is 36 characters including its two spaces. Brightwell Packaging is 20. Turning the first into the second means deleting the last 16 characters. Normalised against the longer string, the similarity is 1 minus 16 over 36, which is 20 over 36, or 0.56.

So the character measure reads 0.56 and the token measure reads 0.67, and the difference is not a matter of calibration. The character measure is answering a question about typing and the pair does not differ by typing. It differs by a word that one system stored and another did not.

That failure gets worse as names get longer, because a dropped word is a larger share of a short name than of a long one under a character measure, while a token measure prices it as one token either way. Character measures earn their place on personal names, where the errors really are keystrokes, and edit distance on surnames is J14's subject.

Cohen, Ravikumar and Fienberg compared the families directly at the IJCAI workshop on Information Integration on the Web in 2003, testing edit distances, token based measures and hybrids on name matching tasks. Their strongest performer combined a token scheme weighted by term frequency with a character comparator applied inside the tokens, which is a good description of what a company name actually needs: word level structure on the outside, tolerance for typing on the inside. Deciding that field by field across a whole registration record, one measure per field, is the argument in J16.

Where plain Jaccard misleads you

Every token counts the same. That is the whole weakness and it produces real false merges.

Northgate Systems Group against Southgate Systems Group. Tokens are northgate, systems, group against southgate, systems, group. Shared tokens are systems and group, and the union is four tokens. The score is 2 over 4, which is 0.50, sitting exactly where most review bands begin, for two firms that share nothing except two of the most common words in any industrial directory.

Count how common. In a directory of 2,400 exhibitors you might find group in 180 company names, systems in 240, packaging in 96 and brightwell in 2. The measure is treating a token that appears in one name in ten as equal evidence to a token that appears in one name in 1,200.

The stopword list is the crude fix and it is worth having. Drop the obvious generics before comparison and Northgate against Southgate falls to zero shared tokens. The trouble is that the list is judgement, it differs by vertical, and in a food ingredients show the word group is generic while in a media portfolio the word group is part of half a dozen genuinely distinct trading names.

The second weakness is length asymmetry, and it catches people who have already fixed the first one. Jaccard divides by the union, so a short name compared against a long one is penalised for words it never had. Kestrel Optics against Kestrel Optics Precision Instruments Manufacturing shares two tokens against a union of five, which is 0.40, even though every token in the shorter name is present in the longer one. If the case you care about is one name being contained in another, which happens constantly when a badge carries the trading name and a contract carries the full registered name, divide by the size of the smaller set instead. That gives 2 over 2, which is 1.0. The two coefficients answer different questions, and running both on the same pair tells you which situation you are in.

Weighting tokens by how rare they are

The better fix uses your own directory to decide what a token is worth. Weight each token by the logarithm of the number of companies divided by the number of company names containing it, which is inverse document frequency, then compute the overlap on weights instead of counts.

Work it on the same two comparisons, using logarithms to base ten.

Brightwell appears in 2 names out of 2,400, so its weight is the log of 1,200, which is 3.08. Packaging appears in 96, so the log of 25, which is 1.40. Systems appears in 240, so the log of 10, which is exactly 1. Group appears in 180, so the log of 13.33, which is 1.13.

Brightwell Packaging Systems against Brightwell Packaging now shares weight 3.08 plus 1.40, which is 4.48, against a union of 3.08 plus 1.40 plus 1, which is 5.48. The weighted score is 4.48 over 5.48, which is 0.82, up from the unweighted 0.67. The pair got more similar because the tokens it shares are the informative ones.

Northgate Systems Group against Southgate Systems Group moves the other way. Say northgate appears in 3 names and southgate in 2, giving weights of 2.90 and 3.08. Shared weight is systems plus group, which is 1 plus 1.13, so 2.13. The union is 2.90 plus 3.08 plus 1 plus 1.13, which is 8.11. The weighted score is 2.13 over 8.11, which is 0.26, down from 0.50.

One threshold at 0.45 now separates the pair you want from the pair you do not, where the unweighted version put them 0.17 apart with both in the awkward middle. Computing the token frequencies is a single pass over the directory you already have, and the resulting weights are a table you can read, argue with and correct, which is a property worth more than it sounds when a sales director asks why two accounts merged.

Where this stops

Token overlap cannot see through an abbreviation. International Business Machines and IBM share no tokens at all and score zero, and so do Deutsche Messe and DMAG, and so does every firm your salespeople refer to by initials. The measure is not wrong, it has no information to work with, and the answer is an alias table rather than a cleverer function.

The second failure is genuine substrings. Brightwell Packaging and Brightwell Packaging Machinery may be two separate legal entities in one group, one selling materials and one selling equipment, and the token measure will score them 0.67 for the same reason it scored the earlier pair 0.67. Nothing in the name distinguishes a shortened version of a firm from a sibling company. That distinction needs the address, the domain or the contracting entity, and the case where one buying group books under four names is J19's problem.

The third is tokenisation itself. Names in scripts without spaces between words, hyphenated German compounds, and the difference between S.A.R.L. and SARL all decide how many tokens exist before the coefficient sees anything. The measure inherits every decision made upstream of it, which is why company name normalisation is its own stage in J17 and not a line of code inside the comparison.

Pull your exhibitor directory into a spreadsheet this week, split the company names into tokens, and count how often each token appears. The twenty most frequent tokens are your stopword list, written by your own file instead of by intuition, and the ratio between the most common and the rarest tells you how much the weighting would change before you implement any of it. That single frequency table is the cheapest useful thing in the whole unified data pipeline.