Company name normalisation before you compare a single exhibitor record

Two rows in an exhibitor directory, both the same refrigeration firm from Hamburg. One reads Moller & Sohne Kalteanlagen G.m.b.H., with umlauts on the first two words. The other, typed by a sales assistant working from a purchase order, reads MOELLER AND SOEHNE KAELTEANLAGEN GMBH.

Count the tokens they share. Zero. Not a low score, an actual zero, because the umlaut version and the expanded version have no character in common at the token level and the punctuation differs on top of that. Every similarity measure you could put on those two strings returns something close to nothing, and no threshold anywhere fixes it.

Company name normalisation is the stage that makes the comparison possible, and it belongs in the pipeline as its own step with its own tests. Christen treats it that way in Data Matching, published by Springer in 2012, where pre-processing sits ahead of indexing and comparison in the sequence of the matching process, on the argument that the quality of what follows depends on it entirely.

The order the steps run in

Normalisation is a sequence, and the sequence is not interchangeable.

Unicode normalisation first. The o with an umlaut can be stored as one code point or as a plain o followed by a combining diaeresis, and those are different byte sequences that display identically. Unicode Standard Annex 15, maintained by the Unicode Consortium and revised in 2025 for Unicode 17.0, defines the four normalisation forms that resolve this, and the point of applying one is that strings which should be equivalent end up with a single binary representation, so a later comparison can be a byte comparison. Skip this step and two records that look identical on screen will fail an exact match test, which is the most infuriating bug in this entire area because you cannot see it.

Case folding second. Lowercase everything, using a case folding operation rather than a naive per character lowercase, since the German sharp s folds to a pair of s characters and a locale aware lowercase in a Turkish locale maps a capital I to a dotless i, which will quietly break every Turkish company name in your file if the server locale ever changes.

Punctuation stripping third. Full stops, commas, hyphens and slashes come out. G.m.b.H. becomes gmbh only after this runs, and Saint-Gobain becomes saint gobain or saintgobain depending on whether you replace punctuation with a space or with nothing, which is a decision you should make once and write down.

Ampersand expansion fourth. Replace the ampersand with the local word for and. This has to happen after punctuation stripping only if your stripping rule leaves the ampersand alone, so most implementations do it first. Either order works. Doing half of each does not.

Whitespace collapse fifth. Multiple spaces, tabs and non breaking spaces become one space, and leading and trailing space goes.

Accent handling last, and twice. Explained below, because it is the step where a single normal form stops being possible.

Legal suffixes come off after all of this, as a separate operation with its own reason, which is J18's subject.

What does the worked case look like?

Run the two Hamburg rows through the sequence.

The first record, Moller & Sohne Kalteanlagen G.m.b.H. with umlauts, normalises to a single code point per accented character, folds to lower case, loses its full stops so that the suffix becomes gmbh, and has its ampersand expanded to and. That leaves five tokens: moller with an umlaut, and, sohne with an umlaut, kalteanlagen with an umlaut, gmbh.

The second record, MOELLER AND SOEHNE KAELTEANLAGEN GMBH, needs less work. Case folding and whitespace collapse give five tokens: moeller, and, soehne, kaelteanlagen, gmbh.

Compare them now and the token overlap is 2 shared tokens out of 8 distinct, so 0.25, from a starting point of zero. The two shared tokens are and and gmbh, which are the two tokens carrying no information at all, so the improvement is illusory and the pair still fails.

Apply the German convention that expands an umlaut into a vowel followed by e, and the first record becomes moeller and soehne kaelteanlagen gmbh. Now all five tokens agree, the score is 5 over 5, which is 1.0, and after the suffix comes off it is 3 over 3, still 1.0. The pair that no measure could link is now an exact match on a derived column.

That is the whole return on the stage. One transformation moved a pair from unreachable to identical, and it did so for every German exhibitor in the file at once.

Why does one normal form not survive transliteration?

Because two conventions are both correct, and they produce different strings.

Add a third row for the same firm, typed by somebody whose keyboard has no umlauts and who simply dropped them: Moller & Sohne Kalteanlagen GmbH, spelled with plain vowels.

Under the expansion rule, the first record becomes moeller and the third becomes moller, so they no longer match. Under the diacritic stripping rule, the first record becomes moller and matches the third, while the second record is still moeller and no longer matches either. One rule links the first to the second. The other links the first to the third. Neither links the second to the third, and there is no third rule that links all of them, because moeller and moller are genuinely different strings once the accent information is gone.

The fix is to stop looking for one normal form. Generate both keys for every record, index the record under each, and treat agreement on any key as agreement. Your storage cost is one extra column. Your candidate set grows by whatever share of your directory contains accented characters, which in a European exhibition portfolio is often 8 to 12 per cent of exhibitor names and almost never more than 20.

This same shape recurs everywhere in international data. A Turkish company name containing a dotless i, a Spanish name with an enye, a Polish name with a slashed l: in each case there is a strict transliteration and a lazy one, and both are in your file because both are in the world. Emitting several keys and taking the union is cheaper and more honest than arguing about which convention is right.

Keeping the original, and keeping the rule version

Never overwrite the raw string. The normalised value is a derived column.

There are two reasons and both bite eventually. The first is commercial: an exhibitor's contract, invoice and badge all carry the name as the exhibitor gave it, and a normalised name on an invoice is an error a finance lead will find in about a day. The second is that you will change these rules. When you add the second accent convention, or decide that hyphens should become spaces after all, you have to recompute every normalised value from the original, and if the original is gone you are recomputing from an already damaged string.

Store the rule version alongside the derived value. When a match looks wrong six months later, the first question is which version of the normalisation produced the key, and that question should take a second to answer, not an afternoon of reading commit history.

Test the normaliser the way you would test any other function, with a fixture file of raw inputs and expected outputs, thirty or forty lines long, drawn from your own directory rather than invented. Every awkward case you meet gets added to it: the firm with an ampersand inside a word, the one whose name is a single accented token, the one with a full stop at the end of a genuine abbreviation. That file is what lets somebody change the ampersand rule in two years without breaking the accent rule, and it takes an hour to build.

Once the derived column exists, the comparison that runs on top of it is a different decision with its own literature, and token overlap on company names in J13 is where that goes.

Where this stops

Normalisation makes strings comparable. It does not make different firms the same firm or the same firm one firm.

The hardest residue is abbreviation. No sequence of case, punctuation and accent rules turns Deutsche Messe into DMAG, or International Business Machines into IBM, because the information is not present in the string. That needs an alias table, maintained by a person, seeded from your own directory by looking at which short names appear in the same postcode as which long ones.

The second residue is translation. A Chinese exhibitor's legal name in Chinese characters and its English trading name share nothing to normalise. Both are correct, both appear on your contracts, and the link between them is a fact you have to store rather than derive.

The third is that normalisation is lossy by design, and the loss occasionally matters. Stripping punctuation makes A.B.C. Systems and ABC Systems identical, which is usually right, and makes Smith & Co and Smithco identical, which is usually right too. It also makes distinct trading names collide in verticals full of short generic names. Track how many raw strings map to each normalised form, and any normal form with more than a handful of distinct raw strings behind it deserves a look before it feeds anything in the unified data pipeline. Deciding which measure to apply per field, which is J16, gets easier once you know how much the normalisation already collapsed.

Take your exhibitor directory this week and count three things: how many company names contain a character outside plain ASCII, how many contain an ampersand, and how many contain a full stop. Those three counts tell you what proportion of your file is currently unmatchable for reasons that have nothing to do with your matching logic, and the fix for all three is an afternoon of string handling.