Building an exhibitor churn prediction model that beats a simple baseline

A data scientist presented an exhibitor churn prediction model to a portfolio review with one number on the slide. Area under the curve, 0.78, on a held out edition. The room took it as good news and moved on to who would work the list.

The question nobody asked was what the same holdout would have produced from a rule anybody could write in SQL in ten minutes. On that portfolio the answer turned out to be 0.63, and the gap between the two numbers, rather than the 0.78, was the only thing in the deck that supported a decision.

Write the baseline down before you build anything

A churn model is a claim that a set of features carries information about renewal beyond what you already know for free. The claim is only testable against a stated alternative, so the alternative has to exist before the model does.

Three baselines are worth having, and they take an afternoon between them.

Everybody renews. The majority class. On a book with 28 per cent churn this is 72 per cent accurate, which is the number that gets quoted in meetings by people who have not thought about it, and its area under the curve is exactly 0.5. Its only job is to kill accuracy as a reporting metric on the first slide.

Persistence. Whatever happened at the last transition happens again. An account that renewed last time renews this time, an account that is new or that has skipped an edition is at risk. This is the honest baseline, because it is what an experienced sales director carries in their head and it costs nothing to compute.

Size rank. Order accounts from smallest stand to largest and call the smallest riskiest. Small exhibitors churn more at every tenure on most shows, so this baseline is often stronger than people expect and it needs one column.

If your model cannot beat persistence by a margin that survives the next section, the interesting finding is that your features carry no information, and that is worth knowing in week two rather than month five.

How should the holdout be cut?

Both the model and the baselines have to be scored on identical rows, and how you cut those rows decides whether the comparison means anything.

Split by edition, not at random. A random split of exhibitor-edition rows puts the 2025 row of an account in training and its 2026 row in test, and the model then learns that account rather than the process. Hold out one complete edition of one show, train on everything strictly before it, and score every method on that edition. It is the closest thing available to the situation the model will face in production, where the future edition genuinely has not happened. What else can bleed backwards through a feature window is a separate audit.

Our worked holdout is one edition of one show: 620 exhibitor accounts, of which 174 churned. The base rate is 174 over 620, which is 28.1 per cent.

The comparison, worked

The persistence baseline flags 168 of the 620 accounts as at risk. Of those 168, 79 actually churned, so 89 flags were wrong. Of the 452 accounts it did not flag, 95 churned.

That gives sensitivity of 79 over 174, which is 45.4 per cent, and specificity of 357 over 446, which is 80.0 per cent. For a rule that outputs one of two values, the area under the curve is the average of those two, so 45.4 plus 80.0 over 2, which is 0.627.

Now the model, a gradient boosted tree on features built from the exhibitor record, scoring 0.71 on the same holdout. Eight points of area under the curve, which sounds decisive.

Convert both into the thing the sales team will actually do, which is work a list from the top. The top decile of 620 accounts is 62 accounts.

The baseline cannot rank inside its flagged group, so the best it can do is hand you 62 accounts drawn from the 168 it flagged, whose churn rate is 79 over 168, which is 47.0 per cent. Expect about 29 churners in those 62. Lift over the base rate is 47.0 divided by 28.1, which is 1.67.

The model's top 62 accounts contain 33 churners, a rate of 53.2 per cent, and a lift of 53.2 divided by 28.1, which is 1.89.

Eight points of area under the curve bought four extra correctly flagged accounts in the top 62.

Widen it to the top three deciles, 186 accounts, because most renewal teams work deeper than a decile. The baseline works all 168 of its flags plus 18 unflagged accounts at their 21.0 per cent churn rate, catching about 83. The model catches about 90. Seven accounts.

Four in the top decile and seven in the top three is the real size of the improvement, and it is a defensible size. What it is not is what 0.63 against 0.71 sounds like when read aloud.

Why is area under the curve the wrong thing to argue about?

Two objections to the metric are worth knowing, and they come from different directions.

Neslin, Gupta, Kamakura, Lu and Mason reported a churn modelling tournament in the Journal of Marketing Research in 2006, in which academics and practitioners took the same data from a public website, built models and submitted predictions scored on two validation databases. They reported accuracy as top decile lift and as the Gini coefficient, and the reason for the first is direct: a retention campaign works the top of the ranked list, so the accuracy that matters commercially is the accuracy in the decile you will call. The two measures also carry different information, because a model can sort the riskiest slice well and sort the safe majority badly, which the Gini coefficient will notice and top decile lift will not.

Hand, writing in Machine Learning in 2009, made the sharper technical objection. Area under the curve is incoherent as a summary of misclassification cost, because the implied distribution of the cost ratio depends on the classifier being scored. Comparing two models by their areas means scoring each on a slightly different metric, which is not what anyone thinks they are doing. He proposed the H measure, which fixes a cost distribution in advance so that both models are judged on the same one.

You do not need to adopt the H measure to take the point. The practical version is that the ranking metric should be chosen from how the list gets worked, and for an exhibitor renewal book that means the number of churners in the top n accounts, where n is the number of accounts a team can call in the window. Report the area under the curve if you like. Argue about the count.

Turning the gap into a deployment decision

The comparison becomes a decision once the counts carry money, which is also the point at which a score is worth wiring into renewal intelligence reporting.

Take the seven extra accounts the model finds in the top three deciles, at an average of 18,400 of space revenue each. That is 128,800 of revenue correctly identified as at risk that the baseline would have missed. What the model returns depends on how many of those accounts an intervention actually saves, which is a separate and considerably harder question, so carry it as an explicit assumption rather than burying it. At an assumed save rate of a quarter, the model contributes roughly 32,200 per edition on this show, and if the same lift holds across six shows in the portfolio, around 193,000 a year against the cost of building, running and maintaining it.

Run the same arithmetic on a model that scores 0.63 against a baseline of 0.627. Its top decile might hold 30 churners against the baseline's 29. One account, 18,400 of revenue, an assumed 4,600 saved per edition. Nobody should build a scoring pipeline, a feature store and a retraining schedule for that, and the honest recommendation is to ship the SQL rule, put the modelling time into the value field, and revisit when there is more history.

My position on where the threshold sits: I want the model to find at least half again as many churners in the worked list as the baseline does, on a holdout of at least 400 accounts, before it justifies the operational weight of a scoring pipeline. On our numbers, 29 to 33 does not clear that, and the case for deploying rests instead on the ranking being usable at all, which the baseline's flat 168 flags are not. That is a real benefit and it should be argued on its own terms rather than smuggled in behind an area under the curve.

Where this stops

The comparison is only as stable as the holdout, and exhibitor holdouts are small.

620 accounts with 174 churners gives a top decile of 62 and a difference of four accounts between the methods. Resample that holdout and the four could easily be two or six. If you want the comparison to survive a challenge, repeat it across every show in the portfolio and every edition you have history for, and report the distribution of the gap rather than one number. A model that beats persistence on nine holdouts out of eleven is a finding. A model that beats it on one is a coin.

The second limit is that the baseline is not fixed. Persistence gets stronger on a stable show and weaker on one that has just changed venue, moved dates or lost an anchor, because the correlation between last edition's outcome and this one is exactly what those events break. So the gap you measured is conditional on a period, and a model that beat persistence by seven accounts in a calm year may beat it by twenty in a disrupted one and by nothing at all when the base rate itself moves.

And none of this addresses whether the accounts the model finds are the accounts worth calling. Ranking by probability of churn ranks by nothing else, and an account at 80 per cent risk holding a nine square metre stand is a worse call than an account at 40 per cent holding 200. The list has to be sorted by expected revenue at risk before anybody dials. A model of this shape also answers one binary question about one edition, so putting a date on the risk instead means a survival model over the same accounts, which is built and reported differently.

Take last edition's holdout this week and compute one line of SQL: the churn rate among accounts that renewed at the previous transition, and the churn rate among accounts that did not. Those two rates give you the persistence baseline's sensitivity, specificity and area under the curve in about five minutes, and every future model conversation at your organisation starts from a number instead of an impression.