Building exhibitor peer group percentiles that a small stand can actually trust

The line in the draft report said 90th percentile. The analyst who wrote it was pleased, because the exhibitor had been unhappy after the previous edition and this looked like good news to lead with.

Exhibitor peer group percentiles are only as trustworthy as the cell they are computed over, and this one had nine companies in it. The exhibitor had come second of nine. One stand pulling out of that category next edition, or one more stand joining it, would have moved the same performance by ten or twelve points, and there was no way for the exhibitor to know that from the sentence as written.

Percentiles are the right output format for exhibitor reporting. They travel well, they hide other exhibitors' raw numbers, and they let a stand manager locate themselves without arithmetic. They also become nonsense faster than almost any other statistic when the population underneath them is small, and the point at which that happens is calculable. Why the comparison happens inside your own floor rather than against a published average is E23's argument and assumed here.

How many percentile points is one rank position worth?

Take n exhibitors in a peer group, sorted by unique leads. The percentile assigned to a stand depends on its rank, and the granularity is fixed by n.

Using the linear interpolation convention that most tools apply by default, a stand ranked i out of n sits at (i minus 1) divided by (n minus 1). So the gap between adjacent ranks is 1 divided by (n minus 1), expressed in percentile points.

At n equal to 6, one rank position is 20 points. At n equal to 9, it is 12.5 points. At n equal to 12, it is 9.1 points. At n equal to 41, it is 2.5 points.

That is the whole design problem in one line. Below about a dozen peers, a single stand moving past you costs double digits on a figure you are printing to two significant figures. The number implies a precision the population cannot support.

Work the 41-peer case to see what good looks like. A stand ranked 28th from the bottom of 41 sits at 27 divided by 40, which is 0.675, so the 68th percentile. Add one competitor who beats it and the figure moves to 27 divided by 41, or 0.659, the 66th percentile. Two points of movement from a change that had nothing to do with the stand. That is tolerable. Twelve points is not.

The cell definition, and why it goes thin

The natural peer group is product category crossed with space band, because those are the two variables that explain most of the difference in lead volume between stands and both are recorded in your contract data already.

Take a show with 780 exhibiting companies, 14 product categories and 4 space bands. That is 56 cells and an average of 13.9 exhibitors per cell, which sounds comfortable until you look at the actual grid.

Exhibitors are not spread evenly across it. Two or three categories hold a third of the floor. The largest space band holds perhaps 8 per cent of companies. The cell for the biggest space band in the smallest category holds two stands, and a percentile over two stands is either 0 or 100.

On a grid like that you should expect somewhere between a third and a half of the cells to fall below any reasonable threshold, and those thin cells will contain a disproportionate share of the exhibitors most likely to complain, because small categories are where the specialist stands live.

The space bands are worth setting deliberately rather than inheriting them from the rate card. Sales bands exist to price space and usually have four or five steps with round boundaries. Analytical bands exist to group stands that face similar conditions, and the boundaries should fall where the floor is thin rather than where the price changes. Sort every exhibitor by contracted square metres and put the cuts at the quartiles of that distribution. On a typical floor that gives something like up to 18, 18 to 36, 36 to 90, and above 90, with roughly 195 companies in each band before the category split. Even bands keep more cells above the threshold than round-number bands do, for the simple reason that they were chosen to.

Do the crossing in that order too. Category first, space band second, because if you have to drop one variable it should be the one that explains less. On most floors that is space, since a 36 square metre software stand and a 36 square metre bearings stand have less in common than two software stands of different sizes.

How many exhibitors do you need in a peer group?

Set a minimum and define what happens when you miss it, once, in writing, before you run anything.

Require at least 12 exhibitors in the cell. If category crossed with space band gives you fewer, fall back to category alone. If category alone still gives you fewer than 12, fall back to space band across the whole floor. If that fails, report a rank in words and no percentile at all.

Twelve is a floor rather than a comfort. At 12 peers, one rank position is still 9.1 points, so the percentile should be presented as a band. Report quartile membership instead of a precise figure: your stand is in the upper quartile of 12 comparable exhibitors. Above about 25 peers, one rank is under 4.2 points and a stated percentile earns its precision.

Whatever the ladder produces, print which rung you landed on. Among the 41 exhibitors in your product category and space band. Among the 63 exhibitors in your product category. That single clause is the difference between a number an exhibitor can weigh and a number they have to trust. If several of these percentiles are then folded into one composite score, the weights doing the folding carry their own argument and that is E22's.

One rule that saves trouble later: the ladder has to be deterministic and it has to be recomputed from scratch each edition. The temptation is to keep an exhibitor on the same rung they were on last time so their two figures are comparable. Resist it. If the category grew from 9 companies to 15, they move up a rung, and the report should say so plainly, because the alternative is a benchmark that quietly means something different for different stands in the same report. Comparability across editions is a real problem and the honest fix is to state the rung and the peer count both times, then let the reader decide whether the two are comparable.

Should the exhibitor themselves be inside their own peer group? Include them. Excluding a stand from its own benchmark shifts the median for large stands in small cells and produces the odd result that a dominant exhibitor is compared against a group that looks nothing like the category as anybody else experiences it. It also makes the peer count differ from the category count, which invites a question you do not want to answer twice.

Which percentile definition you are using

This sounds like a footnote and is not, because your reporting tool and your warehouse probably disagree.

Hyndman and Fan set out nine distinct sample quantile definitions in The American Statistician in 1996, noting that packages frequently use different definitions internally for an explicit quantile calculation than for a boxplot or a probability plot. Their recommendation was the median-unbiased estimator, on the grounds that it holds most of the desirable properties and can be defined without reference to the underlying distribution.

On a peer group of 41 the disagreement between definitions is small. On a peer group of 12 it is not, and a stand can land in a different quartile depending on which convention computed the boundary. Pick one, write it in the methodology note, and make sure the SQL that computes the exhibitor's percentile and the chart that draws the category distribution use the same one. This is the least glamorous part of exhibitor analytics and one of the few that produces visible contradictions inside a single document. Two figures in the same report disagreeing by three points is the sort of thing an exhibitor notices and never fully forgets.

Borrowing strength when the cell is thin

There is a better answer than falling back to a coarser group, and it is worth knowing even if you do not implement it in year one.

Efron and Morris, in the Journal of the American Statistical Association in 1975, worked through what happens when you estimate many related quantities from small samples each. Estimating each one purely from its own data is not the best you can do. Pulling each estimate part of the way towards the overall mean produces lower error across the set, and the amount you pull depends on how noisy the individual sample is.

Applied here: for a cell with 9 exhibitors, do not compute the category median from those 9 alone. Compute a weighted blend of the cell median and the floor median, with the weight on the cell rising as the cell grows. A cell of 9 might sit at 40 per cent cell and 60 per cent floor. A cell of 60 sits at essentially all cell.

The exhibitor's percentile is then taken against a benchmark that is less likely to be an artefact of who happened to book that category this year. It is more work, it needs explaining in the methodology note, and it is the right answer for a portfolio where the same thin categories recur every edition.

The limit

None of this fixes the case where the peer group is well-sized and wrong.

Product category comes from your contract system, and contract categories are sold rather than observed. An exhibitor picks the category that puts them in the aisle they want, and two companies filed under industrial software may be selling to completely different buyers. Your cell has 41 members and a stable percentile, and the comparison inside it is still apples to oranges. The only defence is to look at the category definitions with somebody from sales once a year and merge or split the ones that have drifted.

The second limit is that a percentile carries no information about whether the peer group did well. A stand at the 80th percentile of a category where every member captured almost nothing is being congratulated for the least bad result on a bad floor. Print the peer group median in raw units next to the percentile so that case is visible, and treat a category that came in well below the rest of the floor as E26's diagnosis rather than a ranking problem.

The third is timing. Percentiles computed on day two of a four-day show are computed against a population where half the stands have not started scanning properly, and they will move violently. Compute them after close.

Cross-tabulate your last edition's exhibitor list by product category and space band, and count the cells with fewer than 12 companies. Then count how many exhibitors sit inside those cells. If it is more than one stand in five, decide your fallback ladder before you compute a single percentile, because most of your reports will be using it.