Penalised spline forecasting for pacing curves that only have five editions behind them

The first pacing model I saw at a portfolio exhibitions business fitted a cubic spline to the daily registration curve with knots placed by eye, one at the early bird deadline, one at the launch of the conference programme, one somewhere in the summer. It tracked every closed edition beautifully. Held out against the following year, it was worse than dividing by last year's share.

The two failures were the same failure. Knots placed to explain features you have already seen will explain those features, and a curve with enough freedom to follow every wobble in five closed editions has enough freedom to follow the wobbles that were noise.

Penalised spline forecasting removes the knot decision from the argument and replaces it with one number you can tune against held-out data.

What a penalty buys you that knot placement cannot

The traditional regression spline makes you choose two things: how many knots, and where they go. Both choices are discrete, both are hard to defend in a review, and both change the fitted curve more than anyone is comfortable with.

Eilers and Marx published the alternative in Statistical Science in 1996 and their abstract states the problem and the fix in four sentences. They note that "choosing the optimal number and positions of knots is a complex task", and that equidistant knots give only limited control because their number is small and discrete. Their proposal is to use a large number of knots together with a difference penalty on the coefficients of adjacent B-splines. They call the result a P-spline and show that it connects to the familiar spline penalty on the integral of the squared second derivative.

The consequence for a pacing curve is that you stop arguing about knots. Put forty of them across the registration window at equal spacing, add the penalty, and let one continuous smoothing parameter decide how much structure survives. A continuous parameter can be tuned. A knot at day 63 cannot.

The construction, in the terms Eilers and Marx set out

Lay out the axis as days before doors, from 180 down to 0. Divide it into forty equal intervals. Build cubic B-spline basis functions over those intervals, which gives forty plus three, or 43, of them. Each basis function is a bump covering a few intervals and zero everywhere else, and the fitted curve is a weighted sum of the 43 bumps.

Fitting without a penalty means least squares on 43 coefficients, which against 900 daily observations from five editions sounds like plenty of data. It is not, for reasons I come to below.

The penalty adds a term to the loss: the smoothing parameter multiplied by the sum of squared second differences of the coefficient sequence. A second difference of the coefficients is large wherever three neighbouring bumps disagree about the local slope, so penalising it discourages the curve from changing direction.

Eilers and Marx also point out that the order of the difference is a choice. Second differences are the default and they have a property worth knowing: on data with a linear relationship the second difference penalty has no effect at all, because all the second differences of a linearly increasing coefficient sequence are zero. The penalty is therefore free to be enormous without preventing the curve from having a slope.

What does the penalty do at its two extremes?

Working out both ends of the range is the fastest way to understand what the parameter controls.

Set the smoothing parameter to zero. The penalty vanishes, the fit is ordinary least squares on 43 coefficients, and the effective dimension of the model is 43. Eilers and Marx measure that dimension as the trace of the smoother matrix, following Hastie and Tibshirani, and it is a genuinely useful number to print because it converts a smoothing parameter into a count of parameters that a colleague can reason about.

Now set the smoothing parameter very large with second-order differences. Every second difference is driven to zero, the coefficient sequence becomes a straight line, and the fitted curve becomes a straight line too. The effective dimension is 2.

So the parameter moves the model continuously between a straight line through the campaign and a curve with 43 degrees of freedom, and you get to pick anywhere in between and report the effective dimension you landed on. A fitted pacing curve with an effective dimension of 8 is a defensible object. The same curve described as a cubic spline with forty knots sounds like an overfit and is the same model.

How do you choose the penalty when adjacent days are near duplicates?

Hold out whole editions. This is the single decision that separates a pacing spline that forecasts from one that only describes.

The tempting alternative is standard k-fold cross-validation over the 900 daily observations. Hold out a random 10 per cent, fit on the other 810, score on the 90. It runs in seconds and it is close to worthless here, because each held-out day sits between two retained days one day apart. The model does not have to forecast anything to score well on that. It has to interpolate across a one day gap in a curve that moves by perhaps 40 registrations a day, and a wildly overfitted curve interpolates a one day gap superbly.

Suppose that validation returns a mean absolute error of 41 registrations, which looks excellent, while leaving out a whole edition and predicting its curve from the other four returns 380. Both numbers came from the same fitted model. Only the second one is measuring the thing you will do in production, which is predicting an edition you have never seen.

With five editions the edition-level version gives five folds. That is a small number of folds and it produces a noisy estimate of the optimal smoothing parameter, which is a real cost and is preferable to a precise estimate of the wrong quantity. Plot the held-out error against the log of the smoothing parameter across the five folds and look for a broad flat region rather than a sharp minimum, then take a value from the smoother end of that region. Five folds cannot resolve a sharp minimum and will happily invent one.

A useful cross-check comes from the parametric side. If the penalised fit at your chosen smoothing parameter has an effective dimension of 3, it is spending the same budget as the three parameter logistic that O4 fits, and you should ask what the extra machinery is buying. If it has an effective dimension of 25 and five editions of history, something has gone wrong with the selection.

The knot count question, settled

People still ask how many knots, so it is worth answering with the evidence.

Ruppert took the question up directly in the Journal of Computational and Graphical Statistics in 2002, in a paper on selecting the number of knots for penalised splines. The finding that later work leans on is that the number of knots stops mattering once it is above some minimal level, because the penalty is what controls smoothness. Wang, Shen and Ruppert, writing in the Electronic Journal of Statistics in 2011 on the asymptotics of penalised spline smoothing, cite exactly that result when they establish that a P-spline and a smoothing spline become equivalent provided the number of knots is large enough.

For a 180 day registration window, forty intervals puts a knot every four or five days, which is finer than any feature a registration campaign actually has. Going to eighty makes the fit slower and changes the answer very little. Going to eight puts you back in the regime where knot placement matters and you are choosing where the flexibility lives.

Where this stops

A penalised spline is a description of the curve you have seen, and forecasting is about the part you have not.

Extrapolation is the hard limit. Past the last observed day the basis functions run out, and what the fit does beyond the boundary depends on the penalty order in ways that are not intuitive: with a second difference penalty the extrapolation is linear, which for a cumulative registration curve approaching its close is close to the worst available assumption. The workable pattern is to use the spline for the shape of the observed campaign and a separate, deliberate assumption for the tail.

The second limit is the one hiding in the 900 observations. Those points are not 900 independent measurements of anything. Five editions is the number of independent curves, and every standard error the fitting routine hands you will be far too narrow because it counted days. Treat the effective dimension as a description of flexibility and treat any confidence band produced from the same fit with suspicion.

There is also nothing in a penalised spline that prevents it from dipping. A cumulative registration curve cannot go backwards, and a P-spline fitted to snapshots with a deduplication artefact in them will follow the artefact, so imposing monotonicity on the fit is worth doing at the same time, which is O5's subject. If you plan to stack the fitted curves across editions and look at their common structure, the two-component decomposition in O7 assumes the curves going in are already clean.

Fit the same penalised spline to one show's five editions twice, once selecting the smoothing parameter by random day holdout and once by leaving out whole editions, and write down both chosen values and both held-out errors. If the two smoothing parameters differ by an order of magnitude, you have just measured how much your previous validation was flattering you, and the edition-level number is the one to carry into the rest of your forecasting methods work.