Conformal prediction intervals give coverage without assuming a distribution

The forecast slide says 12,400 registrations at show open, plus or minus 8 per cent. Somebody in the room asks where the 8 per cent came from, and the honest answer is that it came from twice the standard deviation of five residuals, treated as though five residuals told you the shape of a distribution.

Conformal prediction intervals are the alternative worth knowing about, because they give you a band with a coverage guarantee that holds in finite samples and does not depend on the errors being normal, or symmetric, or anything else. Angelopoulos and Bates (2021) set out the split conformal recipe in a form you can implement in an afternoon, and Lei, G'Sell, Rinaldo, Tibshirani and Wasserman (2018) prove the finite-sample result in the Journal of the American Statistical Association.

The catch is real and it arrives early, so it is worth saying now. The guarantee is about a rank, and ranks need enough held-out points to rank. On a portfolio with a handful of editions per show you will be counting them carefully.

What does a conformal prediction interval actually promise?

Split conformal takes your fitted model as a black box. You never touch it. What you do is set aside a calibration set the model has never seen, score every point in it, and use the sorted scores to decide how wide the band has to be.

For a plain regression forecast the score is the absolute error. Forecast the held-out edition, subtract the actual, take the absolute value. Do that for every calibration point and you have n numbers.

Angelopoulos and Bates (2021) give the quantile as "the ceiling of (n+1)(1-alpha) divided by n empirical quantile" of those scores, which in plain terms means sort the n scores ascending and take the one at rank equal to the ceiling of (n+1) times the coverage level. Call that q. The interval for a new point is the point forecast plus or minus q.

The guarantee they state is two-sided: coverage is at least 1 minus alpha and at most 1 minus alpha plus 1/(n+1). At 90 per cent with nine calibration points, that upper bound is 0.9 plus 0.1, which is 1. The interval is allowed to be conservative and with small n it usually is. That is worth knowing before you go to a board with it, because a band that covers 96 per cent of the time while claiming 90 is wider than it needs to be, and somebody will notice it never gets caught out.

The assumption doing the work is exchangeability: the calibration points and the point you are predicting have to be drawable in any order without changing the joint distribution. Independent and identically distributed data satisfies it. A show that has just moved venue does not, and that case belongs to adaptive conformal inference in O22.

The recipe worked on nine held-out editions

Take a portfolio of nine shows, each with enough history to fit whatever model you use. For each show, hold out its most recent edition, fit on everything earlier, forecast the held-out edition, and record the absolute error as a percentage of the actual.

Suppose those nine errors come out, sorted ascending, as 1.8, 2.4, 3.1, 3.9, 4.6, 5.5, 7.2, 9.8 and 14.1 per cent.

For 90 per cent coverage, alpha is 0.1, and the rank is the ceiling of 10 times 0.9, which is 9. The ninth of nine scores is 14.1 per cent. So the interval on a new forecast of 12,400 is 12,400 times 0.141 either side, which is 1,748, giving 10,652 to 14,148.

For 80 per cent coverage the rank is the ceiling of 10 times 0.8, which is 8. The eighth score is 9.8 per cent, and the band is 12,400 times 0.098, which is 1,215, giving 11,185 to 13,615.

Look at what moving from 80 to 90 per cent cost. The half-width went from 1,215 to 1,748, an increase of 44 per cent, and every bit of that came from one calibration point. The 14.1 per cent error belonged to a single show, and it is now setting the width of every interval in the portfolio. This is the honest face of a distribution-free method with small n. There is no smoothing, no fitted tail, nothing to average the outlier away. You get the empirical order statistic you actually have.

Why is nine the smallest set that buys you 90 per cent?

The rank you need is the ceiling of (n+1)(1-alpha), and that rank has to exist inside a set of size n. So you need (n+1)(1-alpha) to be no greater than n.

At 90 per cent: 0.9n plus 0.9 is at most n, so 0.9 is at most 0.1n, so n is at least 9. Eight calibration errors gives you a required rank of 9 in a set of 8, which is undefined, and the honest output is an infinite interval.

At 80 per cent: 0.8n plus 0.8 is at most n, so n is at least 4. Four calibration errors is enough.

At 95 per cent: 0.95n plus 0.95 is at most n, so n is at least 19.

Those three numbers should change how you plan the exercise. A single show with five editions cannot produce a 90 per cent conformal interval on its own under any arrangement of the data. A portfolio of nine shows can, if you are willing to treat one held-out edition per show as exchangeable with the show you are forecasting. A portfolio of nineteen show-editions gets you to 95 per cent. Anything narrower than 80 per cent coverage is available cheaply and is usually the wrong thing to report anyway.

The choice most teams should make is to accept 80 per cent and say so. A four-point calibration set gives a real 80 per cent guarantee. Nine points stretched to claim 95 per cent gives an infinite interval, and quietly dropping to a normal approximation to avoid that is exactly the move that produced the 8 per cent on the slide.

The score you sort decides what the band looks like

Absolute error is the default and it produces a symmetric band. Registration forecasts are usually not symmetric in their errors, because the ways a show overshoots and the ways it undershoots are different mechanisms.

Two changes are worth making. The first is to score the absolute percentage error rather than the absolute count, which is what the worked example above does, because otherwise a 40,000 attendee show and a 3,000 attendee show contribute scores that are not comparable and the big show sets the band for everyone.

The second is to normalise by a predicted spread. Lei and colleagues (2018) describe a locally weighted variant where the score is the absolute error divided by an estimate of the conditional error scale at that point, which gives narrower intervals where the model is reliable and wider ones where it is not. On event data the natural scale estimate is something like the show's own historical year-on-year variability, so a volatile launch-adjacent show gets a wider band than a fifteen-year-old flagship, and the coverage guarantee survives intact because the score is still just a number being ranked.

If you want the band to be asymmetric in a principled way, the score changes again, and the underlying model has to produce quantiles rather than a point. That is conformalized quantile regression, and the quantile fitting half of it is the subject of pinball loss at 0.1 and 0.9 in O23.

What breaks the guarantee

Exchangeability is the whole of it, and event portfolios break it in ways that are easy to miss.

Time order is the obvious one. If your calibration set is the 2021 and 2022 editions and you are forecasting 2026, those points are not exchangeable with the target in any period where the market moved, and the conformal guarantee simply does not apply. It does not degrade gracefully into a slightly-worse guarantee. It is a statement about a distribution you no longer have.

Selection is the subtler one. If you build the calibration set from shows that survived, you have conditioned on survival, and the errors you sorted came from a population that excludes the editions that were cancelled or absorbed. The band you compute is then a band for surviving shows, and using it on a show whose future is uncertain overstates your knowledge in the direction that costs money.

Reusing the calibration set is the third. Every time you look at the calibration errors, tune the model, and recompute, you leak the calibration set into the fit, and the guarantee weakens by an amount nobody can quantify for you. Split it once, write down what you did, and if you must retune, hold out fresh points.

Where this stops

The guarantee is marginal, and that word is carrying more weight than it looks like it is carrying.

Marginal coverage means that across the whole pool of cases the interval contains the truth 90 per cent of the time. It does not mean it contains the truth 90 per cent of the time for your one show. If three of your nine shows are volatile and six are steady, a marginally valid 90 per cent interval can easily cover the steady six about 98 per cent of the time and the volatile three about 70 per cent, and average out at 90.

That is a serious limitation in an event portfolio, because the shows people ask for intervals on are disproportionately the volatile ones. Conditional coverage, meaning 90 per cent for each show separately, is not achievable in a distribution-free way without additional assumptions, and pretending otherwise is worse than reporting the marginal number and saying what it means.

The practical mitigation is to stratify. If you have enough calibration points, build separate conformal quantiles for groups of shows that behave alike, at the cost of a smaller n in each group and therefore a coarser set of achievable coverage levels. With nine points you can reach 90 per cent once. Split into three groups of three and the best guarantee available in each is 75 per cent.

None of this addresses whether the nominal level you started from was defensible, which is a separate question about what a stated interval really covers and belongs to O20.

Start by counting what you actually have. Take your portfolio, list every show-edition you could hold out and score honestly, and see whether that count reaches four, nine or nineteen. That number, before any modelling, tells you the strongest coverage claim you are entitled to make, and it is usually lower than the one currently on the slide. The rest of the forecasting methods work follows from it.