Job title seniority parsing for a global exhibitor and visitor file

A portfolio show with 180,000 registrations over five editions had 41,000 distinct job title strings in it. Head of Category and Category Head were both there, several hundred times each, landing in different buckets. So were Sr. Dir., Global Sourcing and Senior Director Global Sourcing, and a Geschaeftsfuehrer who scored below an assistant buyer because the rule list was written in English.

Job title seniority parsing gets treated as a modelling problem and it is a data cleaning problem with a small classifier at the end. The classifier is the easy part. Deciding what the levels mean, and proving the mapper hits them, is the work.

Why start with rules instead of a model?

Because you have no labels, and a supervised model needs a few thousand of them before it beats a morning of pattern writing.

There is a second reason that matters more in the long run. A rules mapper can be read. When the sales director asks why a category buyer at a large retailer came out as level 4, somebody can point at the rule and change it in an afternoon. When a classifier does the same thing, the answer is a retraining cycle and a shrug, and the rule that caused the problem is spread across a weight matrix.

Rules also fit the shape of the data. The distribution of title strings has a very short head and a very long tail. In the file above, the 500 most frequent distinct strings covered 104,000 of the 180,000 rows, which is 57.8 per cent, and those 500 can be checked by hand in a day. The tail is where a model would earn its keep, and the tail is 3 per cent of your hosted buyers.

Five levels, defined by decision rights

The levels have to be defined before any parsing happens, and they should be defined by what the person can do rather than by which words appear in their title.

• Level 1, executive. Owns or runs the business unit. Founder, chief officer, managing director, general manager of a company.
• Level 2, director. Owns a function with a budget. Vice president, director, senior director.
• Level 3, head. Runs a category, a region or a team, usually with recommendation rights and a spending limit.
• Level 4, manager. Manages work or people, specifies requirements, rarely signs.
• Level 5, practitioner. Does the work. Engineers, designers, buyers without a mandate, analysts, students.

Five is a deliberate choice. Standard occupational schemes go much finer, and the International Labour Organization's ISCO-08, published in 2008, is a four level hierarchy in which managers occupy Major Group 1 and skill level is applied below the major group rather than across it. That structure is built for national statistics, and it is more granularity than a match score can use, since authority fit in I14 will compress whatever you give it into a handful of bands anyway.

Normalise before you match

Every parser that fails does so because it matched raw strings. The pipeline that works has four stages and none of them is clever.

Lowercase and strip punctuation, which collapses Sr. Dir. and sr dir. Expand a fixed abbreviation table: sr to senior, jr to junior, svp to senior vice president, vp to vice president, gm to general manager, dir to director, mgr to manager. Split compound titles on commas, slashes, ampersands and the word and, then keep the highest seniority fragment, because Head of Sales and Marketing Manager is one person at one level and it is the higher one.

Then match on token sets. This is the step that fixes Head of Category against Category Head, because the set of significant tokens is identical once the stop word of is dropped. Prefix rules and regular expressions anchored at the start of the string cannot do this without a rule per word order, and there are more word orders than you think.

Order the rules from most specific to least specific and stop at the first match. Assistant to the chief executive has to be caught before chief executive, or your file fills up with level 1 assistants. Deputy, interim and acting all need a decision written down, and the defensible one is that a deputy sits one level below the title they deputise for.

How do you know the mapper works?

Hand label a sample and report the result per level. The shared task literature settled the format for this a long time ago: Tjong Kim Sang and De Meulder's introduction to the CoNLL-2003 shared task on named entity recognition, run over English and German data, held out a labelled test set and scored systems with precision, recall and F-measure computed per entity type, so a system could not hide a weak class behind a strong one.

The same discipline applies to a title mapper, and it is worth the day it takes.

Draw 500 titles and label them by hand. Draw them by row frequency, since a title appearing 4,000 times matters 4,000 times more than a singleton. Then run the mapper and compare.

Suppose the labelled sample contains 62 level 1 titles, 88 level 2, 71 level 3, 154 level 4 and 125 level 5. The mapper gets 58 of the level 1 titles right, which is 93.5 per cent. Level 2, 74 of 88, which is 84.1 per cent. Level 3, 43 of 71, which is 60.6 per cent. Level 4, 139 of 154, which is 90.3 per cent. Level 5, 118 of 125, which is 94.4 per cent.

Total correct is 58 plus 74 plus 43 plus 139 plus 118, which is 432 out of 500, or 86.4 per cent. That number would go in a status update and it would be misleading, because the average of the five per-level rates is 84.6 per cent and one level is at 60.6 per cent. Level 3 is where head, lead, principal and senior manager collide, and it is the level your hosted buyer programme is mostly made of.

Report both averages every time. The gap between them is the thing to manage.

The queue that catches these failures is a permanent part of the system. Sort unmatched strings by row count each week and the top of that list is the next twenty rules, which takes an hour and moves coverage by a point or two at a time. Track two numbers alongside the accuracy figures: the share of rows landing at unknown, and the share of rows matched by the last catch-all rule in your list, since a catch-all that fires on 9 per cent of rows is doing more work than any individual pattern and nobody has looked at what is in it.

Languages, and the abbreviations that mean two things

Translation destroys seniority, so translate nothing and write rules per language instead.

The head of a non-English distribution is short. In a European visitor file, a few dozen strings cover most of the German, French, Italian and Spanish rows: geschaeftsfuehrer, prokurist, directeur general, responsable achats, direttore, responsabile acquisti, gerente, jefe de compras. Each one maps cleanly once somebody who speaks the language spends an hour on it, and that hour is cheaper than any automated approach available.

Two traps are worth naming. Some words are false friends across languages, and responsable in French is closer to level 3 or 4 than to level 2, while director in Spanish often sits higher than director in American English. And some abbreviations are genuinely ambiguous: GM is general manager at most firms and grocery manager in food retail, which is a level 1 versus level 4 error in exactly the vertical where it will be noticed.

Where a rule cannot decide, the right output is unknown. An unknown level should fall back to a neutral value in the score, in the same way a missing category does, and it should appear on a queue so that repeated unknowns become new rules. A parser that guesses when it cannot tell produces confident errors, and confident errors are the ones that reach an exhibitor.

Where this stops

A title is a claim about a job, and people write whatever they like on a registration form. Founder appears on badges belonging to two person consultancies and to companies with 400 staff. Some organisations inflate titles as compensation, some deflate them as culture, and Japanese and German firms use grades that do not map cleanly onto an Anglo-American ladder at all.

The mapper cannot fix any of that, and neither can a better model, because the information is missing at source. What the mapper can do is be consistent, which means the same string always produces the same level and every level is measured. Consistency is enough for a score, because the score is comparing candidates within your own file.

The second limit is that seniority is one person's ladder, and the purchase usually is not. A level 4 manager who represents an account with a large committee behind them is worth more than the parsed level says, which is the buying committee problem in I16.

This week, pull the 100 most frequent title strings in your registration file with their row counts, and label them by hand into five levels. Count what share of rows those 100 strings cover. If it is over half, which it usually is, you have most of a working mapper in a spreadsheet before you write any code, and the same technique that maps free text profiles onto categories in I8 will handle the tail when you get to it, feeding the matchmaking score the levels it has been guessing at.