Skip to content

Leave one year out validation leaks information on a growing show

Forecasting methodsUpdated 2026-08-238 min read

In short

Leave one year out validation trains on editions that came after the one being predicted, so the model sees the trend and the market conditions it is supposed to forecast. Measured error falls and operational error does not. Bergmeir, Hyndman and Koo (2018) give the narrow conditions under which cross-validation stays valid.

The slide says the model achieves 3.2 per cent mean absolute percentage error across eight shows and forty editions, validated by leave one year out cross-validation. Forty folds, one held-out edition each, all shows covered.

It is a good-looking number and it describes a task nobody in the building will ever be asked to do. Leave one year out validation trains on editions that came after the one being predicted. When you forecast the 2026 edition in real life, 2027 has not happened. When the validation forecast the 2023 edition, 2024 and 2025 were sitting in the training set telling it roughly where 2023 had to land.

The bias is not small and it is not random. On a growing show it runs in one direction, and it is largest for exactly the shows whose behaviour changed, which are the shows anyone actually needs a forecast for.

What exactly leaks when you hold out a middle year?

Two things leak, and they are worth separating because the fixes differ.

The first is trend. A show growing steadily produces a nearly straight line, and predicting an interior point of a straight line from the points either side of it is interpolation. Predicting the next point off the end is extrapolation. Interpolation is a much easier problem, and a model evaluated on it will report an error that has almost no relationship to what happens when the same model is pointed forward.

The second is the calendar-year effect, and on a portfolio this one is bigger. Suppose 2023 was a strong year across your whole sector. A model with any portfolio-level term will estimate that 2023 effect from the other shows' 2023 editions, and then apply it to the held-out show. The validation forecast of show A's 2023 edition therefore knows what 2023 was like, because seven other shows in the training set were also in 2023.

Operationally you never know that. Forecasting 2026 in August 2026 means the 2026 portfolio effect is unobserved, and the model has to set it to zero or guess it. Every bit of the accuracy the validation earned from knowing the year effect is accuracy you will not have.

The portfolio year effect, worked

Put numbers on the second leak, because it is the one people argue about.

Take eight shows and five editions each, so 40 rows. Fit a model with a show-level level and a portfolio-wide year effect. Suppose the estimated year effects, expressed as a percentage above or below the portfolio trend, come out as minus 12, plus 2, plus 9, minus 3 and plus 4 across 2021 to 2025.

The mean absolute year effect is 12 plus 2 plus 9 plus 3 plus 4, all divided by 5, which is 30 over 5, giving 6 per cent.

Under leave one year out, the model predicting show A's 2023 edition estimates the 2023 effect from seven other observations of it, and gets something close to the plus 9. The residual it is left to explain is small.

Under a forward forecast of 2026, there are no observations of the 2026 effect at all. The best the model can do is set it to the mean of past effects, which is zero by construction, and absorb the whole thing into the error.

So the validation error and the operational error differ by roughly the size of the year effect, which here is about 6 percentage points. A validated 3.2 per cent becomes something closer to 9 per cent in use, and the difference is not model failure. The difference is the validation having answered a different question.

Four folds in five are contaminated

The contamination is not uniform across folds, and counting which ones are clean is a useful exercise.

With five editions per show, leave one year out gives five folds. Holding out 2025 is clean, because everything in the training set genuinely precedes it. Holding out 2024, 2023, 2022 or 2021 is not, because each of those training sets contains at least one later edition.

Four folds in five are contaminated. The reported average is a blend of one honest number and four flattering ones, weighted four to one in favour of the flattering ones, and no amount of averaging separates them.

That ratio gets worse as the series lengthens, which is counterintuitive. Ten editions gives nine contaminated folds out of ten. The only clean fold is always the last one, and the last one is the fixed holdout you were trying to improve on.

Contamination is also graded rather than binary. Holding out 2024 leaves exactly one later edition in the training set, so the interpolation is barely constrained on one side. Holding out 2022 leaves three later editions, and the held-out point is surrounded. If you rank the five folds by how many later editions sit in their training set, the reported error usually falls as that count rises, and seeing that pattern in your own fold-level results is confirmation that the leak is operating rather than a theoretical worry.

There is a related trap in how the folds get used. If you also select the model specification by comparing leave one year out scores, the contamination compounds, because the selection now depends on the leaked information too. That interacts badly with how easily a five-point series is overfitted, which is O12's subject.

When is cross-validation legitimate on a time series?

The blanket rule that cross-validation must never touch a time series is wrong, and the exception is precise enough to be useful.

Bergmeir, Hyndman and Koo (2018), in Computational Statistics and Data Analysis, show that "in the case of a purely autoregressive model, the use of standard K-fold CV is possible as long as the models considered have uncorrelated errors." They note that such a setup occurs, for example, when the models nest a more appropriate model, which is common when machine learning methods are used, and they argue cross-validation is particularly suitable there for controlling overfitting.

The conditions matter. Purely autoregressive means the predictors are lagged values of the series itself and nothing else. Uncorrelated errors means the residuals carry no remaining serial structure, which is testable and frequently fails on event data because of the calendar-year effects described above.

Bergmeir and Benitez (2012), in Information Sciences, had earlier argued for a blocked form of cross-validation as the standard procedure for time series, using all available information while avoiding the theoretical problems that come from temporal dependence. Blocking keeps contiguous stretches together instead of scattering neighbouring observations across folds.

For an event portfolio, the honest reading is this. If your model is purely autoregressive on one show's own history, and you have checked the residuals for serial correlation, cross-validation is defensible and it recovers folds you otherwise could not use. If your model has portfolio-level terms, calendar-year terms, or any predictor measured contemporaneously across shows, cross-validation leaks and no amount of blocking within a single show fixes it, because the leak runs across shows rather than across time.

Measuring the size of your own leak

The good news is that the leak is measurable with data you already have, and the measurement takes about an hour.

Run the same model twice on the same forty rows. Once under leave one year out. Once under a strictly forward rolling origin, which for a five-edition series means training through 2023 and forecasting 2024, then training through 2024 and forecasting 2025. That gives you sixteen forward test points across eight shows, and the design of that second run is O25's subject.

Compare the two error figures. The gap is your leak, in the units you already report. If leave one year out says 3.2 per cent and the forward run says 8.7 per cent, you have found 5.5 percentage points of measured accuracy that came from information the model will not have next year.

Do not treat the gap as noise to be argued about. With eight shows and two forward origins each you have sixteen forward errors, which is enough to look at the distribution of the difference rather than only its mean, and whether a difference of that size is more than sampling variation is a question with its own answer in O27.

Where this stops

Removing the leak costs you test points, and on a five-edition series you cannot afford many.

Leave one year out gave forty folds. The forward alternative gives sixteen, and if your model needs four editions to fit rather than three it gives eight. That is the trade, and it is a real one: an unbiased estimate from eight observations has a wide spread, and a biased estimate from forty has a narrow one that is centred in the wrong place. Preferring the first is a judgement about which failure mode does more damage, and my view is that a confidently wrong accuracy claim does more damage than an honestly imprecise one, because the first gets designed into a commitment and the second gets treated with the caution it deserves.

There is one situation where the argument genuinely goes the other way. If you are choosing between two model families rather than estimating an accuracy figure to publish, and both families suffer the same leak in the same direction, the comparison can still be informative even though neither number is usable on its own. Say that out loud when you do it, and never let the leaked figure escape into a board pack.

Start by counting your folds. Open whatever validation your team currently runs, list which held-out editions had a later edition of the same show or a same-year edition of a sibling show in their training set, and write the fraction on the slide next to the accuracy number. On most portfolio setups it will be four in five or worse, and that fraction is the beginning of every honest conversation about the rest of your forecasting methods.

Questions people ask about leave one year out validation

Why is leave one year out validation biased on a trending series?
Because the training set for an interior fold contains later editions than the held-out one. A model fitted on 2022 and 2024 is interpolating when it predicts 2023, and interpolation is an easier problem than the extrapolation you face operationally. The measured error describes a task nobody will ever perform.
Is cross-validation ever valid for time series?
Bergmeir, Hyndman and Koo (2018) show that standard K-fold cross-validation is valid for a purely autoregressive model provided the models considered have uncorrelated errors, a condition that holds when the model nests a more appropriate one. Bergmeir and Benitez (2012) recommend a blocked form of cross-validation as the general default.
How do you measure how much your validation leaks?
Run both schemes on the same model and the same data. Compute the error under leave one year out and under a strictly forward rolling origin. The gap between the two is the leak, expressed in the same units as your accuracy metric, and it is the amount by which the first scheme has been flattering you.

Related reading

All forecasting methods articles