Match score calibration and why raw scores mislead your concierge team

A concierge manager works out how many proposals to send. The buyer has a twelve meeting quota, the top of their queue is scoring around 0.72, so twelve divided by 0.72 gives about seventeen proposals and that is what goes out. Three weeks later the buyer has eight confirmed meetings and everybody is confused.

Match score calibration is the thing that was missing. The 0.72 was a ranking position wearing the costume of a probability. On last edition's file, proposals in that band were accepted 44 per cent of the time, so seventeen proposals were always going to produce seven or eight meetings, and the arithmetic that produced seventeen was reasonable given a number that was not.

What does a match score of 0.72 promise?

Under most implementations, nothing. The score is a weighted sum of five bounded sub-scores, and a weighted sum of numbers between zero and one is a number between zero and one. That is a scale coincidence rather than a probability.

The distinction matters the moment anybody divides by the score. Capacity planning divides by it. Quota filling divides by it. A concierge team telling an exhibitor that a proposal is high confidence is making an implicit claim about frequency. Every one of those uses needs the score to mean what it looks like it means, and checking whether it does takes an afternoon.

The check has a name and a shape. Take proposals whose outcome you know, group them by predicted score, and compare the average prediction in each group against the share of that group that was accepted. A calibrated score sits on the diagonal. An overconfident one sits below it everywhere.

Bucketing last edition into deciles

Take 4,000 proposals from a completed edition, each with a predicted score at the time it was shown and an outcome of accepted or declined. Sort by predicted score and cut into ten buckets of 400.

Deciles are a compromise. Fewer buckets hide the shape, more buckets give you 40 proposals per bucket and a wobble you will over-read. With 400 per bucket, the standard error on an observed share near 0.30 is the square root of 0.30 times 0.70 divided by 400, which is 0.023, so roughly two and a half points. Any gap smaller than about five points is inside the noise.

The outcome definition needs the same care it needs anywhere. Accepted and held is the event. Accepted and no-show is a different failure and mixing it in makes the score look worse than it is at the job you asked it to do.

One more split is worth running before you fit anything. Calibrate hosted buyers and self-service visitors separately, because the two groups face completely different costs when they accept. A hosted buyer whose flights and hotel are paid against a meeting quota accepts at a much higher rate than a visitor deciding on the Sunday whether to give up 30 minutes of a day trip, and pooling the two produces a curve that is right for neither. If the split leaves either group under 1,000 proposals, run the pooled curve and say in the caption which group dominates it.

Two numbers summarise that table. The mean predicted score across all 4,000 proposals is 0.451, and the overall acceptance rate is 0.272, so the system is overconfident by 17.9 points on average. The mean absolute gap across the ten buckets, which is the expected calibration error, is also 0.179 because the buckets are equal in size.

Why do well ranked scores end up badly calibrated?

Look down the table again. Acceptance rises from 6 per cent in the bottom decile to 55 per cent in the top one, without a single inversion. The ordering is excellent. The scale is wrong everywhere.

That combination is the normal case rather than a curiosity. Niculescu-Mizil and Caruana showed at ICML in 2005 that several widely used learning methods produce predictions that rank well while sitting a long way off the diagonal, with characteristic distortions that differ by method, and that a post hoc correction fixes the scale without touching the ordering.

A hand-weighted match score has its own reason for landing high. The sub-scores measure suitability, and suitability is only one of the things standing between a proposal and a meeting. The buyer has a quota of twelve and receives thirty proposals. The buyer is travelling on the Tuesday. The exhibitor's slots are full by the time the request arrives. None of those appear in product fit or authority fit, and all of them cap the acceptance rate below what suitability alone would predict.

Fitting the correction with isotonic regression

The correction you want is a monotone function from predicted score to acceptance probability. Monotone matters because it preserves your ranking, so nothing downstream reorders.

Isotonic regression fits exactly that: a step function that never decreases, chosen to minimise squared error against the observed outcomes. Zadrozny and Elkan set out its use for turning classifier scores into probability estimates at ACM SIGKDD in 2002, and the mechanics are simple enough to run in a spreadsheet with the pool adjacent violators algorithm.

On the table above there is nothing to pool, because the observed shares already rise at every step. The fitted mapping is then the decile column itself: a proposal scoring 0.72 gets mapped to 0.44, and one scoring 0.85 gets mapped to 0.55.

Pooling starts to matter when a decile inverts. Suppose decile 7 came back at 0.33 and decile 8 at 0.31. The algorithm merges the two into one block at their weighted average, which with equal bucket sizes is 0.32, and both deciles map to 0.32. That is the whole method: walk left to right, and wherever the next value is lower than the current one, merge them and take the average, repeating until the sequence never falls.

Fit on one edition and check on the next. A mapping fitted and evaluated on the same file will look perfect and tell you nothing, and with 400 proposals per bucket the fitted values carry a two to three point error band that a second file will expose.

What calibration changes on the concierge desk

Go back to the buyer with the twelve meeting quota. Under the corrected mapping, proposals at the top of their queue carry an acceptance probability of 0.44, so filling the quota needs twelve divided by 0.44, which is 27.3, so 28 proposals. The original seventeen was short by eleven.

That arithmetic scales into the capacity plan for the whole programme. 200 hosted buyers on a twelve meeting quota need 2,400 held meetings, and at 0.44 acceptance that is 5,455 proposals to send. At the raw 0.72 the same plan looks like 3,333 proposals, and the programme would finish 1,000 meetings short with nobody able to say why. Whether the floor can physically hold 2,400 meetings is a separate question and belongs to I32.

Calibration also changes what the team is allowed to say. A recalibrated 0.44 supports the sentence that fewer than half of these proposals usually get taken up. A raw 0.72 supports a sentence about high confidence that the file does not back, and exhibitors remember which one they were told.

There is a third use, and it is the one that pays for the afternoon. Corrected scores add up. If you sum the corrected probabilities across every outstanding proposal for one buyer, you get an expected number of meetings for that buyer, and summing across buyers gives an expected total for the programme with three weeks still to run. Twenty-eight outstanding proposals averaging 0.38 gives an expectation of 10.6 meetings against a quota of twelve, which is a shortfall you can act on while there is time. The same sum computed from raw scores would have read 20.2 and told you everything was fine. Nobody can do that arithmetic with numbers that are not probabilities, which is the practical reason to stop treating a ranking score as one.

Where calibration stops

A calibrated score is a statement about a group. Among proposals near 0.44, roughly 44 per cent get accepted. It says nothing about which ones, and no amount of calibration work improves the ordering, because the correction is monotone by construction. If your ranking is poor, calibration makes the poor ranking honest and leaves it poor.

Calibration also decays. It is fitted on one edition and applied to the next, and the acceptance rate moves when you change the quota, the proposal deadline, the number of proposals per buyer or the size of the exhibitor list. Any of those shifts the whole curve, and a mapping fitted before a quota change is measuring a system that no longer exists. Refit each edition, keep the previous mapping, and report the change between them.

The last limit is thin data. A first edition has no outcomes to calibrate against, and borrowing a mapping from a sister event assumes the two programmes have similar acceptance behaviour, which is I19's argument and not something to assume quietly. Below roughly 1,000 proposals with known outcomes, decile buckets get too small to fit anything and the honest move is to report the raw score with the overall acceptance rate printed next to it.

Start by pulling last edition's proposals into a two column file: predicted score and accepted yes or no. Sort, cut into ten buckets, and compute the mean predicted and observed share in each. If the gap in the top decile is larger than five points, every capacity number your team calculated from those scores is wrong in a known direction, and fixing the mapping is cheaper than fixing what the sub-scores contain in I1 or re-running the weighting argument in I2. Attach the corrected value to the same record as the reason code that led the proposal, which is I5's subject, so your matchmaking reporting can show both together.