Churn model label leakage and the fields that encode the answer

The first exhibitor churn model I ever saw presented internally scored 0.89 on a holdout. The room was pleased. Somebody in finance asked what the second-best feature was, mostly to be polite, and the answer was opportunity stage.

Opportunity stage is a field a salesperson updates during the renewal conversation. The model had been shown, for every account, a summary of how the renewal conversation went, and asked to predict whether the renewal happened. It scored 0.89 because it was reading the answer off the back of the page.

Churn model label leakage is the single most common failure in renewal modelling and it is almost never caught by the person who built the model, because every check they run comes back clean. Cross-validation is clean. The holdout is clean. The learning curve looks healthy. The model is genuinely excellent at the task it was given, and the task was wrong.

Leakage has a formal definition and it is worth using

Kaufman, Rosset, Perlich and Stitelman gave the treatment in ACM Transactions on Knowledge Discovery from Data in 2012. Their definition is the introduction of information about the target that would not legitimately be available at prediction time, and their prescription is what they call learn-predict separation: every feature must be computable from data that existed at the moment you would actually need to score the account.

They point at the controversies around the INFORMS 2010 Data Mining Challenge and the IJCNN 2011 Social Network Challenge as evidence that the problem is very much live. Public competitions with prize money and thousands of eyes on the data did not catch it in advance. That should adjust anyone's confidence about catching it in an internal project with one analyst and a deadline.

The phrase worth carrying into an exhibition context is the moment you would actually need to score the account. Write it on the wall. Every argument about whether a field is legitimate reduces to that question.

The fields that leak in an exhibitions CRM

Exhibition renewal data has a fairly consistent set of offenders, and most of them are respectable-looking columns that live in the same table as the legitimate ones.

Opportunity stage, or renewal status. Populated by a rep as the conversation progresses. Anything past "contacted" is a partial readout of the outcome. Stage is a legitimate input to a weighted pipeline forecast, which is a different job with its own calibration problems (G34), and it has no place in a risk score computed before anyone has been contacted.

Contract issued, contract sent date, contract status. A contract only gets issued to an account that is renewing. This one is close to a perfect predictor and is completely useless, because at scoring time no contract has been issued to anyone.

Deposit or first invoice raised for the next edition. Same problem, one step further along.

Rebooking portal login. A login to the rebooking form is an act of renewing. If your engagement features are computed over a window that includes the rebooking period, this is inside them and you may not notice, because the feature is called something bland like portal_sessions_90d.

Next edition stand number or floor plan allocation. Operations assigns space to confirmed exhibitors. If your feature build joins to the floor plan table without a date filter, the join itself is the leak.

Lost reason code. Obviously the answer, and it still gets into feature sets, usually because somebody selected all columns from the account table.

Account owner changed. Subtler. Accounts get reassigned when they lapse and go to the win-back team, so a recent owner change on a churned account may be a consequence of churn recorded before your snapshot date.

The common shape is that all of these are written after the renewal conversation began. That is the test, and it is easier to apply than trying to reason about each field's semantics.

Where should you cut the feature window?

The fix is structural. Pick a cutoff date, use only data that existed on that date, and rebuild every feature against it.

For an annual exhibition the natural cutoff is show close, because that is the last moment before the renewal process starts and it is the moment when a risk score would be operationally useful. Scoring on the Monday after the hall empties gives the sales team the whole cycle to act.

Doing this properly means the feature build has to be time-aware rather than a set of joins against current-state tables. Every aggregate needs a date filter. Every status field needs to be sourced from a history table if you have one, and dropped if you do not, because a current-state status column tells you the value today and not the value at show close.

Most teams discover at this point that they have no history table for the fields they care about, which is a real finding worth reporting. It is also the cheapest thing to fix going forward: start snapshotting the account table on the day each show closes, and in three editions you will have the data you should have had all along.

What the leak cost, in money

The recovery is unpleasant to present, so it helps to have the arithmetic ready.

Take a portfolio with 2,400 exhibitor accounts, a base churn rate of 28 per cent, and an average account value of 14,500. The retention programme contacts the top decile by predicted risk, 240 accounts, at a fully loaded cost of 1,000 per account including the incentive, so 240,000. Assume 30 per cent of contacted churners are saved.

The leaky model scored 0.89 with a top-decile lift of 3.1. That decile would contain 0.28 times 3.1 times 240, which is 208 churners. At a 30 per cent save rate that is 62 accounts retained, worth 899,000, against a cost of 240,000. Net 659,000. That is the number that went in the business case.

Rebuilt with the feature window cut at show close, the model scores 0.71 with a top-decile lift of 2.4. The decile now contains 0.28 times 2.4 times 240, which is 161 churners. Saves 48 accounts, worth 696,000, against the same 240,000. Net 456,000.

The gap is 203,000 of promised value that was never available. Nobody stole it and nobody lied. It was an artefact of scoring the model on information the model would not have when it ran.

Verbeke and colleagues argued in the European Journal of Operational Research in 2012 for exactly this style of evaluation, computing the maximum profit achievable from acting on a churn model and using that to select the model and the fraction of the base to contact. Their point was that a model chosen on a statistical score alone is often not the model that makes the most money. The corollary for leakage is sharper: once you evaluate in money, a leak stops being an abstract methodological error and turns into a specific overstatement in a document with somebody's name on it.

How do you test a model for leakage?

Sort features by importance and read the top five aloud. If any of them describes something a human does during the renewal process, you have found it. This catches the majority of cases and takes two minutes. Do the sorting with permutation importance on a holdout if you can, because the default chart in a tree ensemble flatters continuous columns and will bury a binary leak below the noise (G11).

Check the AUC against the calendar. Score the model using only data available at show close, then at show close plus thirty days, then plus sixty. A model whose performance climbs steeply as the window extends into the renewal period is being fed the outcome incrementally. A clean model improves gently or not at all.

Look for suspicious separation on a single feature. Plot the churn rate by decile of each candidate feature. A feature where the top decile is 96 per cent churn and the bottom is 2 per cent is either a leak or the answer to a different question, and in eight years of looking at these it has never once been a legitimate signal.

Where this stops

Cutting the window at show close removes the leaks you can name. It does not make the model honest about the thing the model is silently assuming, which is that next year looks like last year.

There is also a leak that no date filter catches, and it is worth naming because renewal teams create it themselves. If your account managers have been running a manual risk list for three editions, and the accounts on that list got extra attention, then the outcomes in your training data have already been shaped by the intervention. The model learns the pattern of accounts that survived after being rescued, and the pattern of accounts that were never flagged. Neither is a clean estimate of unassisted risk. There is a whole post's worth of feedback loop in that and it belongs elsewhere (G39), but the honest position is that a churn model fitted on a base your own team has been actively working is measuring something more complicated than churn.

The narrower limit is that 0.71 may simply be what your data supports. A drop from 0.89 to 0.71 feels like a failure and it is a correction. The question after a rebuild is whether 0.71 beats whatever your team is doing now, which is a comparison against a naive baseline and a separate piece of work (G10). Both numbers belong in the same renewal intelligence record, dated, so nobody quotes the old one by accident.

Start by opening your current model's feature list and marking every column with the name of the person or process that writes it and the point in the cycle it gets written. Anything written by a salesperson, by operations after show close, or by finance during the renewal cycle comes out. Refit with what is left and note the new score before anyone else sees the old one.