Which exhibitor churn model features carry real signal in exhibitions

The chart has sixty bars on it. Somebody has sorted them descending, coloured them blue, and put it on slide four. The sales director looks at it for about six seconds and asks the only question that matters, which is which of these am I supposed to watch.

Nobody has a good answer. The chart came out of whatever feature_importances_ returned, and nobody has asked which exhibitor churn model features are stable across years, whether half of them are the same fact recorded twice, or whether the top one is a column that only exists because the renewal already happened.

Choosing features for exhibitor churn is a smaller and more tractable problem than the sixty-bar chart suggests. Most shows have four or five families of data worth building from, the families are ranked in a fairly consistent order, and the ranking can be produced honestly in an afternoon.

Start where the data actually is

The instinct is to reach for engagement data first. Email opens, portal sessions, webinar attendance, content downloads. It is the modern-looking answer and for most exhibition businesses it is the wrong place to start, because the engagement record is thin, recently instrumented, and inconsistent across editions in a way that makes any model built on it a model of your marketing automation rollout.

Billing data does not have that problem. Every exhibitor who took space has an invoice, and the invoice exists for every edition back to whenever finance last changed systems. It is complete, it is consistent, and it encodes more about the state of a commercial relationship than most people expect.

The 2022 study in Applied Sciences on customer churn prediction in B2B non-contractual settings is the clearest demonstration of how far this gets you. The authors built their entire feature set from invoice-level records, 280,502 invoices covering 3,470 customers over 38 months, with no behavioural or engagement data at all, and tested three standard classifiers against several different definitions of churn. Their conclusion is that workable models for each of those definitions can be derived from invoice-level data alone. The point worth taking is that a usable churn model existed before anyone connected a marketing platform to anything.

An exhibition has a structural advantage over the distributor in that study, because your customers buy on a fixed annual cycle. You know exactly when the purchase decision happens. That turns a messy non-contractual problem into something close to a discrete renewal event, which is easier.

Which five families of features are worth building?

Everything I would build first comes out of the invoice ledger, the floor plan and the registration system.

Recency, frequency and monetary value across editions. The three classic quantities, computed in editions instead of days. Editions since last exhibited, count of editions exhibited out of editions run, and total invoiced value at the last edition. Frequency does most of the work here. An exhibitor on their seventh consecutive edition behaves nothing like one on their second.

Space trend. Square metres held this edition divided by square metres held last edition, and the same figure over two editions back. This is the single feature most account managers already watch informally, and it deserves to be in the model as a ratio, not a difference, because a 6 square metre cut on a 12 metre stand means something different from a 6 metre cut on a 240 metre one.

Days from invoice to payment. Computed per invoice, then summarised per account per edition as a median, and then differenced across editions. The absolute level tells you about the exhibitor's accounts payable process. The change tells you about their finances, and the change is what predicts.

Service order behaviour. Count of orders placed through the exhibitor portal, normalised by space, plus whether anything was ordered at all. An exhibitor who takes 90 square metres and orders nothing beyond the shell scheme has either brought their own contractor or has stopped investing in the stand, and the second reading is common enough to be worth a feature.

Badge allocation used. Every exhibitor gets an entitlement of exhibitor badges. The proportion they actually register against that entitlement is close to free to compute and it measures how much of their own staff time they are prepared to spend on your show.

Nine or ten columns from those five families is a reasonable starting set, and it is the whole of the feature layer most renewal intelligence work rests on. If your model needs sixty, something has gone wrong upstream, usually one-hot encoding of a category field with forty levels.

How do you rank features by permutation importance?

The importance number that ships with tree ensembles by default is impurity-based, and it is biased. Strobl and colleagues showed this clearly in BMC Bioinformatics in 2007: impurity importance inflates variables with more possible split points, so continuous features and high-cardinality categories look more important than they are, purely as an artefact of how trees choose splits. In an exhibitor file this systematically flatters days-to-payment and systematically buries binary flags.

Permutation importance avoids that. Fit the model, score it on a holdout, then shuffle one column at a time in the holdout and score again. The drop in performance is that column's importance. It measures what the fitted model actually uses on data it has not seen.

Take a show with five editions of history and 1,840 exhibitor-edition rows, of which 28 per cent churned. A gradient boosted model reaches 0.742 area under the curve on the holdout. Whether 0.742 is any good is a different question, settled by comparing it against a naive baseline, and that sits with G10. Shuffling each column twenty times and averaging the result gives this:

The drops sum to 0.116. The top two account for 0.061 of that, which is 53 per cent. Two features carry half the discriminating power of the model, and the bottom three carry effectively none.

That table is the answer to the sales director's question, and it took one afternoon and no new data collection.

The second filter is stability, not size

A single ranking on a single holdout is a sample, and it will move. The filter that matters more is whether a feature keeps its rank when you refit on a different pair of editions.

Refit the model three times, predicting the 2023 renewal from 2022 data, the 2024 renewal from 2023, and the 2025 renewal from 2024. Record each feature's rank in each fit. Days-to-payment change ranking 1, 2, 1 is a feature you can build a process around. Service orders per 100 square metres ranking 5, 2, 9 is not, and the honest reading is that one edition had a service disruption that made the column briefly predictive.

The cut I would use is blunt: keep a feature if its worst rank across three refits is inside the top half of the set, drop it otherwise. This throws away some genuine signal. It also throws away the features that will make the model look stupid in front of a sales team when the ranking flips next year, and given that the model's value depends entirely on whether anyone acts on it, that trade is worth making.

I would go further and say the sizing is close to irrelevant below a threshold. A feature contributing 0.004 of AUC contributes nothing a rep can act on, and it costs a pipeline step, a data quality dependency and a line in the documentation that somebody will have to maintain for years. Keep six features you can explain in a sentence each.

Features that look strong and are not

Three patterns come up repeatedly and are worth naming.

The first is the same fact twice. Space in square metres, space in square feet, space band, and stand type are one feature wearing four hats. Correlated duplicates split the importance between themselves, so each looks moderate, and the family as a whole looks less important than it is. Collapse them before you rank anything.

The second is a proxy for size. Total invoiced value, contact count, badge entitlement and service order count all correlate with how big the exhibitor is. If several of your top features are size proxies, your model may be learning that big exhibitors renew, which is true, already known, and not actionable. Test it by fitting within size bands and seeing whether the ranking survives.

The third is a field that only gets populated during the renewal conversation. That is label leakage and it has its own post (G13). The tell is a feature that appears from nowhere at rank one with an importance far above the rest.

Where this stops

Permutation importance answers a narrow question: which columns is this particular fitted model relying on. It does not answer which factors cause exhibitors to leave, and the two get confused constantly in the meeting after the chart goes up. It also says nothing about when an account will go, which is what survival models are for and belongs to G12.

Days-to-payment does not make an exhibitor churn. A budget review inside the exhibitor's business makes them pay late and makes them churn, and your model sees the payment because you have a payment record and does not see the budget review because nobody sent you the minutes. Acting on the feature by chasing payment faster addresses none of it.

There is also a correlated-feature failure mode that permutation importance does not escape. If two columns carry the same information, shuffling either one leaves the model able to recover most of the signal from the other, so both show small drops and both look unimportant. The honest fix is to permute correlated groups together and report the group's importance, which is more work and gives a more defensible number.

None of that argues against building the table. It argues for reading it as a description of the model and then going and asking three account managers what they think is behind the top two features, because their answer is the causal hypothesis you cannot get from the data.

Start this week by pulling one show's invoice ledger for the last three editions and computing a single column: median days from invoice to payment per account per edition, then the change between the two most recent editions. Sort your current exhibitor list by it descending and read the top twenty names. If more than a handful of them are accounts your team is already worried about, the feature works on your show and the rest of the model is worth building.