Uplift modelling for renewal offers finds the exhibitors worth persuading

Here is a result that should bother anyone running a renewal programme. A mobile operator ran a churn reduction campaign and the treated customers left at a higher rate than the untreated ones. Churn went from 9 per cent to 10 per cent. The campaign was actively losing customers, and without a control group nobody would have known, because 10 per cent churn among the people you targeted for being at risk looks like a success.

Radcliffe and Surry report that case in their 2011 Stochastic Solutions white paper on uplift modelling, published as Portrait Technical Report TR-2011-1. The interesting half is what happened next. An uplift model identified a subsegment of about 30 per cent of the population where the intervention did help. Targeting only that subsegment brought overall churn from 9 per cent to under 8 per cent while cutting campaign spend by 70 per cent. Same offer, same customers, different targeting rule, and the sign of the effect flipped.

Renewal outreach in an exhibition business has exactly the same structure, and most renewal analytics work stops at a churn score with no control group behind it.

What is the renewal call actually doing?

When you phone an exhibitor about next year, four things can happen and only one of them is the reason you called.

Some exhibitors were going to renew regardless. The call is friendly and costs a slot. Some were never going to renew, and the call is a polite twenty minutes confirming it. Some are genuinely undecided, and a well-timed conversation with a good floor plan attached moves them. And some were drifting towards a renewal they had not thought hard about, and your call is the thing that makes them think hard about it.

That fourth group is real in exhibitions and people underestimate how real. An exhibitor who has renewed at the same 36 square metres for six editions without a review is one phone call away from asking their marketing lead whether the show is still worth it. If the honest answer is marginal, you have started a procurement conversation you would rather not have started, and by the time it finishes they have taken a call from the competing show in the same category.

A churn model cannot see any of this, because it predicts what will happen and not what your call changes. The two are different quantities and the difference is the entire point.

Uplift is a difference, so you have to create the difference

Uplift for an account is the renewal probability if you contact them minus the renewal probability if you do not. You never observe both for the same account, which is the standard problem in causal inference, and the standard answer is to randomise so the two groups differ only by the treatment.

Practically: before the renewal window opens, take the book, randomly assign a control group, and do not call them. No rep outreach, no personalised email, nothing beyond whatever every exhibitor gets. Log the assignment somewhere the sales team cannot quietly override, because they will, and one determined rep working eleven control accounts because they know them personally ruins the estimate for the whole cycle.

Then measure. Take 2,000 accounts, hold out 400 at random, contact the other 1,600. Suppose 71 per cent of the contacted group renews and 66 per cent of the control group does. Overall uplift is 5 percentage points. On 1,600 contacted accounts that is 80 extra renewals, which at an average 18,000 per account and a 45 per cent contribution margin is 80 times 18,000 times 0.45, or 648,000 of contribution.

That number alone justifies the programme. It also hides everything useful, because 5 points is an average across accounts whose true uplifts range from strongly positive to negative.

Modelling the difference

The obvious approach is to build two churn models, one on the treated accounts and one on the control accounts, and subtract. It works badly. Both models carry error, the difference between them carries both errors, and the quantity you want is usually much smaller than either prediction, so the noise swamps the signal. I have seen this produce uplift rankings that are essentially random and look plausible.

The better approach models the difference directly. Radcliffe and Surry's significance-based uplift trees split the data on the variable that most significantly separates treatment response from control response, so the tree grows towards heterogeneity in the effect instead of heterogeneity in the outcome.

Athey and Imbens, in the Proceedings of the National Academy of Sciences in 2016, gave the version with proper statistical guarantees. Their causal trees partition the sample into subgroups with different treatment effects, and their central contribution is what they call honest estimation: use one sample to decide the splits and a completely separate sample to estimate the effect inside each leaf. Without that split, the same data both chooses the subgroup that looks most responsive and measures how responsive it is, which is a machine for producing subgroups that do not replicate. Anyone who has watched a "high response segment" from last cycle evaporate this cycle has seen the failure that honest estimation exists to prevent.

What comes out, and what to do with it

Score the book, sort by predicted uplift, and read it in deciles. That ordering replaces the expected-revenue ordering the at risk queue starts from, which is G22's subject. The shape you are looking for is monotone and it usually is not perfectly monotone.

Take the same 2,000 accounts. In the top uplift decile, 74 per cent of contacted accounts renew against 58 per cent of control, a 16 point effect. In the middle deciles it thins to 4 or 5 points. In the bottom decile, 61 per cent of contacted accounts renew against 68 per cent of control, which is a 7 point negative effect on the accounts you were most confident about calling.

Now the allocation changes, inside whatever slot capacity the team has that cycle, which is G23's calculation. If you call only the top four deciles, 800 accounts, at an average uplift of say 10 points, you get 80 extra renewals from half the calls. Same 80 renewals as calling everyone, roughly, because the middle deciles were adding little and the bottom decile was subtracting. That is the same trade Radcliffe and Surry describe, arrived at with a smaller book.

The bottom decile deserves a decision rather than a shrug. I would not leave those accounts in silence, because silence has its own risk. I would move them to a low-salience path: the floor plan opens, the invoice arrives, the stand is held, nobody asks them to justify the decision. Turning a renewal into an administrative default is a legitimate strategy and it is the opposite of a persuasion call.

How big does the book have to be?

Here is why most single shows cannot do this, and it is arithmetic rather than pessimism.

To detect a 5 percentage point difference between two proportions near 0.68, at 80 per cent power and a 5 per cent two-sided significance level, the rough requirement is 16 times p times one minus p, divided by the squared difference. That is 16 times 0.68 times 0.32, which is 3.48, divided by 0.0025, giving about 1,392 accounts per arm. Near 2,800 accounts in total to detect the average effect once.

Detecting heterogeneity in that effect, which is what an uplift model estimates, needs considerably more, because you are estimating a separate difference inside each leaf. A show with 600 exhibitors cannot do this in one edition. It can do it across five editions if the treatment is applied consistently, and a portfolio of eight shows can do it in one cycle if the renewal motion is similar enough across them to pool.

If you are below that, the honest position is to run the holdout anyway and report the overall uplift with its confidence interval, and to skip the model. An organiser who knows their renewal calling is worth 5 points, plus or minus 6, knows something valuable, which is that they do not yet know whether it works.

Where this stops

The control group is a real cost and it is not only the lost renewals. Reps hate it, and a rep who has been told not to call an account they have known for nine years experiences the experiment as a policy that damages a relationship. Some of that is manageable with framing and some of it is a genuine conflict between learning and this quarter's number.

The second limit is that uplift estimated on one offer tells you nothing about another. If the treatment was a rep call with an early-bird deadline, the model tells you who responds to a rep call with an early-bird deadline. Change it to a discount and you have a different treatment with a different heterogeneity pattern, and the sleeping accounts you learned to leave alone might respond well to a floor plan preview and badly to a phone call. Every distinct offer needs its own arm, which multiplies the sample requirement above.

The third is the one people forget. Uplift models are trained on treated and untreated accounts from a period when the treatment was assigned randomly. The moment you start using the model, assignment stops being random, and next cycle's data cannot be used to retrain it unless you keep a permanent random holdout. That is one instance of the feedback loop any scored sales book develops, which G39 treats properly. Budget for that holdout forever, at maybe 10 per cent, and treat it as the cost of knowing whether the programme still works.

Before the next renewal window opens, take 10 per cent of the book at random, flag it in the CRM as held out, and tell the sales director it is off limits. One cycle later you will have the first honest estimate of what your renewal calling is worth, which is a prerequisite for every other decision in this area.