Overfitting short time series is easy when every model fits five points

The chart looks convincing. Five editions of registration data, a fitted curve running through all five almost exactly, an R squared of 0.99 in the corner, and a forecast for next year that the show director is about to put in a budget.

Overfitting short time series is the normal outcome here, and the tell is precisely that the curve fits so well. Five annual observations cannot contradict a flexible model, so they do not, and the model returns the compliment by extrapolating whatever wiggle it found.

Why does a quadratic look so good on five points?

Take a real-shaped series. Registrations across five editions of 11,800, 12,600, 12,900, 14,200 and 15,900. Growth, with the third edition flat and the fifth strong.

Fit a straight line by least squares. The slope is 980 registrations per edition, the fitted values are 11,520, 12,500, 13,480, 14,460 and 15,440, and the residual sum of squares is 704,000. Against a total sum of squares of 10,308,000, that is an R squared of 0.932.

Now add a squared term. The fitted values become 11,920, 12,300, 13,080, 14,260 and 15,840, and the residual sum of squares falls to 144,000. R squared goes to 0.986. On the chart the curve now passes within a couple of hundred registrations of every point, and the largest residual has dropped from 580 to 300.

Nothing suspicious has happened yet. A quadratic has three coefficients and you gave it five points, so of course it fits better. The problem is what it does next.

The linear forecast for edition six is 16,420. The quadratic forecast is 17,820. For edition seven the linear model says 17,400 and the quadratic says 20,200. By edition eight the line says 18,380 and the curve says 22,980, a gap of 4,600 registrations, all of it produced by a curvature parameter estimated from five numbers with two residual degrees of freedom.

The extra 4,600 is not a demand signal. It is the second difference of a five-point series, projected forward three times.

What the information criteria say, and why they disagree

The usual defence is that a penalised criterion will catch this. It does not, and the arithmetic is short enough to check.

Hyndman and Athanasopoulos give the regression form in Forecasting: Principles and Practice, third edition (2021), as AIC equals T log of SSE over T, plus 2 times k plus 2, where T is the number of observations and k the number of predictors. Run it on the three candidates.

The mean-only model has SSE of 10,308,000, so SSE over T is 2,061,600, its natural log is 14.54, and the AIC is five times that plus four, which is 76.7. The linear model has SSE of 704,000, giving 11.86 and an AIC of 65.3. The quadratic has SSE of 144,000, giving 10.27 and an AIC of 59.3.

Plain AIC prefers the quadratic, and it will keep preferring the more flexible model at this length because two extra units of penalty cannot compete with a log likelihood that improves every time you add a term.

The corrected version is supposed to fix that. The same book gives it as AIC plus 2 times k plus 2 times k plus 3, all divided by T minus k minus 3, and notes that "for small values of T, the AIC tends to select too many predictors". For the mean-only model the correction is 12 divided by 2, which is 6, giving 82.7. For the linear model it is 24 divided by 1, which is 24, giving 89.3. For the quadratic, T minus k minus 3 is 5 minus 2 minus 3, which is zero, and the criterion does not exist.

So on five observations the uncorrected criterion picks the quadratic, the corrected one cannot evaluate the quadratic at all, and among the two models it can evaluate it picks the flat mean of 13,480 that nobody in the room will accept. Two criteria, two answers, one of them undefined. Anyone presenting a model choice justified by AIC on a five-point series is presenting an artefact of the formula.

Counting what each candidate wants to estimate before you fit it is a better first filter, and the parameter budget five observations leaves you is O11's subject.

Does holding out an edition help when you only get one a year?

Yes, and it is the only thing in this post that actually discriminates. It also feels wasteful, because throwing away one of five observations sounds expensive, and it is worth being clear that you are not throwing anything away permanently. You fit on a prefix, score the next edition, then refit on everything for the forecast you publish.

Five editions give two honest origins. Fit on editions one to three and predict edition four. Fit on editions one to four and predict edition five. The mechanics of doing this properly, and the traps in rolling the origin forward on a growing series, belong to O25 and O26.

The five numbers, held out

Three candidates, two origins, absolute errors in registrations.

Carrying the last edition forward predicts 12,900 for edition four against an actual of 14,200, an error of 1,300, then 14,200 for edition five against 15,900, an error of 1,700. Mean absolute error 1,500.

The straight line fitted on the first three editions has a slope of 550 and forecasts 13,533, an error of 667. Refitted on the first four its slope is 750 and it forecasts 14,750, an error of 1,150. Mean absolute error 909.

The quadratic fitted on three points passes through all three exactly, and its second difference of minus 500 turns the series downward. It forecasts 12,700 for edition four, an error of 1,500, worse than doing nothing. Refitted on four points it forecasts 15,375, an error of 525, its best moment. Mean absolute error 1,013.

The ordering has reversed. On fit, the quadratic won by a distance. On held-out error, the line beats it by about 10 per cent and beats the last-edition-carried-forward benchmark by nearly 40 per cent. Two error pairs is thin evidence, and comparing methods on four or fewer paired errors has its own treatment in O27, but thin evidence pointing the right way beats a goodness-of-fit statistic pointing the wrong way.

What the literature says about reaching for the complicated one

Green and Armstrong reviewed this question across the forecasting literature for the Journal of Business Research in 2015. They found 97 comparisons of simple against complex methods in 32 papers, and reported that none of the papers provided a balance of evidence that complexity improves forecast accuracy. In the 25 papers with quantitative comparisons, complexity increased forecast error by 27 per cent on average.

That is a finding about forecasting generally, on series far longer than five points. On a five-point series the case is stronger, because the mechanism that makes complexity expensive, which is variance in the estimated parameters, is at its worst when there is almost nothing to estimate from.

Their explanation for why complexity persists is worth reading in full. Part of it is that complex methods can be used to produce forecasts that support a plan somebody already has, which is a failure mode any show team will recognise from a budget round.

What to fit instead

A level and a slope, fitted on the show's own five editions, is usually the whole model. Report the slope with its standard error so the room can see how firm it is.

If that feels thin, the productive move is to add data instead of parameters. Eight shows with five editions each is forty observations, and a model that estimates one shared growth pattern with a per-show deviation spends far fewer parameters per show than eight separate curves do. That is borrowing strength across shows, which O14 covers, and it is the honest answer to a short series in a portfolio business.

The second addition that costs nothing is within-year data. A show with five editions has five annual totals and perhaps two hundred weekly registration snapshots. Any method that forecasts the final number from the shape of the campaign is fitting to a much longer series, and the parameters it estimates are supported by observations rather than by hope.

Where this stops

Held-out error on two origins is a weak instrument, and it will sometimes pick the wrong model. With two comparisons and errors in the high hundreds, a difference of 100 registrations between candidates is inside the noise, and the correct reading is that the two are indistinguishable on this evidence.

There is also a failure mode that held-out testing cannot see. If the show changed in a way that makes the past unrepresentative, a venue move or a date shift, then every candidate is being scored against a target the data cannot describe, and the model with the lowest held-out error is the one that happened to guess best about a change none of them knew about. That is a structural break problem and it sits with O31 and O32.

The honest position after all this is that the difference between a good and a bad model on five annual observations is smaller than the difference between a well-specified and a badly-specified question. Whatever you fit, the interval matters more than the point, and how badly a nominal 95 per cent interval undercovers on five editions is O20's subject.

This week, take one show's five final numbers, fit the line and the curve, and write both forecasts for three editions ahead on the same sheet of paper. If the gap between them at edition eight is larger than any year-on-year change the show has ever had, you have measured your own overfitting, and the number in the middle of that gap deserves less confidence than either model claims. The wider argument about which methods a portfolio can support sits with the forecasting methods behind it.