Match reason codes give your team something to argue with

Somebody asks, in the review after the show, whether the matching worked. The concierge lead says the product matches went well and the ones based on seniority did not, which sounds right to everybody in the room and is supported by nothing. Two people remember it differently. The meeting moves on.

Match reason codes exist so that conversation ends with a number instead. Every proposal carries a short label from a fixed list saying which part of the score drove it, that label sits on the record next to the outcome, and after the show you count. It is a small piece of instrumentation and it changes what the review can conclude.

Why can't free text explanations be counted?

Because free text is written once and read never. A concierge manager typing a reason into a box produces a sentence for the buyer and nothing for the organisation.

Try it with a hundred proposals from your last edition. Strong product match. Both in packaging. Buyer looking for exactly this. Category overlap plus buyer is a director. Those four describe two distinct situations, and no grouping you can do afterwards recovers which. Adding a language model to categorise the free text after the fact rebuilds the code list you should have had, and does it with an error rate nobody measures.

Herlocker, Konstan and Riedl showed at ACM CSCW in 2000 that the form an explanation takes changes how compelling users find the recommendation, testing explanation interfaces experimentally rather than assuming. Gedikli, Jannach and Ge extended the comparison in the International Journal of Human-Computer Studies in 2014, measuring several explanation types against efficiency and satisfaction. If format changes the outcome, format is a variable, and a variable you never record is one you can never learn from.

What a reason code list looks like

Short, mutually exclusive at the lead position, and written in the vocabulary your team already uses.

Eight codes covers most programmes. The list has to be mutually exclusive at the lead position, which is enforced by the rule that the lead code is the one attached to the largest weighted contribution, so ties get broken by a fixed order and never by whoever wrote the query.

Two codes on that list are worth arguing about before you build it. EXHIBITOR_REQUEST records a commercial decision, and mixing it into a list of fit reasons means your acceptance analysis will be partly measuring your sales team. Keep it, and keep it flagged. BEHAVIOUR_PRIOR sits on data the buyer does not know you hold, which is the exposure question in what you show exhibitors and belongs to I6.

Logging which code led

Three fields on the proposal record. The lead code, the full ordered list of codes that fired, and the version of the code list in force.

Storing only the lead code loses too much. A proposal where product and intent both fired is a different animal from one where product fired alone, and the second field costs nothing. Storing the version matters because you will change the list, and comparing acceptance by code across two editions with different lists is the kind of mistake that survives for years.

The sentence the buyer reads is generated from the lead code and its template, which keeps the buyer-facing text tied to the same arithmetic as the ranking. That mechanism is I4's subject and this post assumes it.

One rule prevents most of the trouble: the code is written by the system at the moment the proposal is created, never by a person afterwards. A concierge manager reviewing a queue on the Thursday and back-filling codes from memory produces a column that looks complete and measures the manager. If your team overrides proposals by hand, and most teams do for their most valuable buyers, log the override as its own code and count it separately. A hand-placed meeting is chosen on grounds the score never saw, usually a commercial relationship, so folding it into the same acceptance figures makes both numbers harder to read.

Reading the distribution against outcomes

Take an edition with 2,400 proposals and 717 accepted, an overall acceptance rate of 29.9 per cent.

PRODUCT_EXACT led 900 proposals and 315 were accepted, which is 35.0 per cent, a lift of 1.17 against the base. INTENT_SAVED led 400 and 180 were accepted, which is 45.0 per cent, a lift of 1.51 and the strongest reason on the list. PRODUCT_ADJACENT led 600 with 132 accepted, 22.0 per cent, a lift of 0.74. AUTHORITY_SENIOR led 300 with 60 accepted, 20.0 per cent. BEHAVIOUR_PRIOR led 200 with 30 accepted, 15.0 per cent, half the base rate.

Check the error band before anyone acts. On 200 proposals, an observed rate of 15 per cent carries a standard error of the square root of 0.15 times 0.85 divided by 200, which is 0.025, so two and a half points. The 15 per cent is really somewhere around 10 to 20 per cent, which is still clearly below the base rate. On a code with 60 proposals the band would be five points wide and the finding would be noise.

The instructive line is INTENT_SAVED at 45 per cent off 400 proposals. If saved-interest signals accept at half again the base rate and lead only one proposal in six, the ranking is under-using the strongest thing it knows, which is an argument for revisiting the weights in I2 with this number in hand.

Before acting on that, check the obvious confound. INTENT_SAVED can only lead a proposal for a buyer who has saved something, and a buyer who has saved something has already come back to the site at least twice. Some of that 45 per cent is the signal and some of it is the fact that engaged buyers accept more of everything. The cheap test is to compute the acceptance rate of PRODUCT_EXACT proposals among buyers who saved at least one exhibitor and compare it against PRODUCT_EXACT proposals among buyers who saved none. If the first group sits at 41 per cent and the second at 31 per cent, then roughly ten of the fifteen points separating INTENT_SAVED from the base rate belongs to the buyer rather than the reason, and the case for reweighting is much weaker than the headline suggested.

What does a low performing code tell you?

Usually that the match was thin everywhere else.

AUTHORITY_SENIOR leads a proposal only when authority is the largest weighted contribution, which under weights of 0.30 on product and 0.15 on authority requires product fit to be under half of authority fit. A senior buyer with weak product overlap is exactly the proposal an exhibitor complains about, and the code is doing its job by telling you these exist and that 300 of them went out.

That reframes what to do about a weak code. The fix is not to remove the code, since removing it changes the label rather than the proposal. The fix is a floor: no proposal goes out where product fit is under 0.3, whatever the total says. Apply that rule to the file above and roughly 300 proposals disappear, the base acceptance rate rises to around 33 per cent because the removed group accepted at 20, and the buyers who were receiving them get 300 slots back for proposals with something behind them.

BEHAVIOUR_PRIOR at 15 per cent says something different. Meeting the same category again is a weak reason to meet, and buyers apparently know it.

Versioning the list

Change the list once a year, between editions, and never mid-edition.

Every change needs a mapping from old codes to new ones, or an explicit statement that no mapping exists. Splitting PRODUCT_EXACT into an exact leaf match and a match on a synonym is a genuine improvement and it makes every prior year's PRODUCT_EXACT figure incomparable, which is fine as long as the chart says so.

Keep the list in the same file as the weights, owned by the same person, reviewed at the same time. The two are the same policy seen from different ends.

Where reason codes stop

A code records why the system proposed the meeting. It says nothing about why the buyer accepted, and those differ more than they should. A buyer accepting because the exhibitor is a household name will show in your data as an acceptance under PRODUCT_EXACT, and your lift figure will quietly include brand recognition you cannot separate out.

The counting also has a selection problem baked in. You only observe outcomes for proposals you made, so a code that never leads a proposal has no acceptance rate, and the codes that look good are partly good because your current ranking chose them. Comparing lifts across codes is honest. Concluding that a code with no proposals would perform badly is not.

The last limit is that a lead code is one number standing in for five. Two proposals both led by PRODUCT_EXACT can have wildly different totals, and averaging their outcomes hides that. Carry the score alongside the code, and read them together.

Start by adding two columns to the proposals table for the next edition: lead code and code list version. Populate the lead code from the largest weighted contribution, which needs no new model and no new data. After the show, group by lead code and compute the acceptance rate against the overall rate, ignoring any code with fewer than 150 proposals. The first version of that table is usually enough to settle an argument your matchmaking review has been having for three years.