Rolling origin evaluation for a show that produces one number a year

Somebody presents a forecasting model and says it was 96 per cent accurate on the held-out year. One year was held out. One forecast was made. One error was measured, and it came to 4 per cent.

That figure is a sample of size one, and the confidence attached to it in the room is not available anywhere in the arithmetic. Rolling origin evaluation is the standard fix, and on an annual show it is both more valuable and more constrained than it is anywhere else, because the whole series is five numbers long.

Tashman (2000) reviewed the design in the International Journal of Forecasting, working through series-splitting rules, fixed against rolling origins, updating against recalibration of coefficients, fixed against rolling windows, and single against multiple test periods. Every one of those choices bites harder when the series is short.

What does rolling the origin actually give you?

Hyndman and Athanasopoulos (2021) describe the procedure in the third edition of "Forecasting: Principles and Practice" as a series of test sets, each consisting of a single observation, where the corresponding training set consists only of observations that occurred prior to the observation that forms the test set. They note that the approach is sometimes known as "evaluation on a rolling forecasting origin" because the origin at which the forecast is based rolls forward in time, and that forecast accuracy is computed by averaging over the test sets.

The reason this matters more than it sounds is that a fixed holdout gives you one draw from the distribution of errors, and one draw tells you almost nothing about the spread of that distribution. Rolling the origin gives you several draws, each produced by a model that never saw its own test point.

The count is easy. Let m be the smallest number of editions your model needs to fit at all, and T the number of editions you have. The number of test points is T minus m. That subtraction is the entire budget, and no amount of clever resampling increases it, which is the connection to the parameter budget five observations leaves you in O11.

Two test points from five editions, worked

Take a show with editions from 2021 to 2025 and a pacing model that needs three editions to estimate its curve.

Fit on 2021, 2022 and 2023. Forecast 2024. The final registered figure comes in at 11,240 against a forecast of 10,780, so the error is 460, which is 4.1 per cent of the actual.

Refit on 2021 through 2024. Forecast 2025. The actual is 12,050 against a forecast of 13,210, so the error is 1,160, which is 9.6 per cent.

Two errors: 4.1 per cent and 9.6 per cent. The mean absolute percentage error is 6.85 per cent.

Report both numbers, and put the mean second. The two errors differ by a factor of more than two and they point in opposite directions, one under and one over. A single figure of 6.85 per cent conceals both facts, and both facts are what a show director needs when deciding how much weight to put on next year's output.

Compare that to what a fixed holdout would have produced. Hold out 2025 alone and you report 9.6 per cent. Hold out 2024 alone and you report 4.1 per cent. The same model, the same data, and two accuracy claims that would support two different decisions. Tashman's review is largely a catalogue of ways that this happens, and the defence is to stop treating any single split as informative.

Rolling the origin inside the booking window

The second axis is the one that rescues an annual series, and it comes from noticing that a show does not produce one number a year. It produces a cumulative registration count every day, and a forecast of the final figure can be made from any of them.

So roll the origin twice. Roll it across editions, as above, and within each held-out edition roll it across lead times: forecast the final figure from 150 days before doors, then 120, then 90, then 60.

With two held-out editions and four lead times that is eight forecast-and-outcome pairs. The temptation is to average all eight and report one number, and that is the wrong move, because the eight are not eight independent observations. Four of them describe the same edition at different distances from the same outcome and are heavily correlated with each other.

Score by lead time instead. At 150 days out you have two errors. At 60 days out you have two errors. What you get is a profile, and the profile is the useful artefact: a table of four rows showing how the error shrinks as the window closes. If the 150-day error is 14 per cent and the 60-day error is 5 per cent, you now know something operational, which is that commitments needing better than 10 per cent accuracy cannot be made before about 100 days out.

That profile is also the thing to compare across shows and across years, and it degrades gracefully. Losing an edition costs you two rows of a four-row profile rather than half of your entire evidence base.

Scale it across a portfolio and the arithmetic starts to work in your favour. Eight shows, two held-out editions each, four lead times, gives 64 forecast-and-outcome pairs, of which 16 sit at each lead time. Sixteen errors at 90 days out is enough to look at a distribution rather than a mean, and the shows are far closer to independent of each other than the lead times within one show are. That is the argument for evaluating a portfolio's forecasting process as one exercise even when each show is forecast separately.

Should the window expand or stay fixed after a change?

Tashman (2000) separates two ways of advancing the origin. An expanding window keeps every observation before the origin, so the training set grows. A fixed window keeps a constant number of recent observations and drops the oldest as the origin moves.

On five editions the expanding window is almost always right, because a fixed window of, say, three editions throws away 40 per cent of a data set that was already too small. The exception is a show that genuinely changed: a move from a two-day to a three-day format, or a merger with a competing event. There the older editions are measuring something else, and including them biases every origin.

The practical compromise is to run both, compare the error profiles, and keep the one that wins on the later origins rather than on the average. A fixed window that beats the expanding window only at the last origin is telling you a change happened recently, which is information worth having independently of which evaluation you adopt.

Be honest about what that comparison costs. Running two window schemes over two origins gives you four numbers, and picking the winner from four numbers is a selection made on very thin evidence. If the two profiles are within a percentage point of each other, keep the expanding window, because it is the simpler procedure and the one that will still work when a sixth edition arrives.

What you must not do between origins

Tashman also distinguishes updating the coefficients of a fixed model specification from recalibrating the specification itself. On short annual series the distinction is the difference between an honest evaluation and a fictional one.

Re-estimating the same model's parameters at each origin is correct and is the whole point. Choosing which model to use at each origin, by looking at how each candidate performed, is not, because the candidate set was selected using information from the test points. The measured accuracy then describes a procedure that included you looking at the answers.

The rule that survives contact with a five-edition series is to fix the specification before the first origin, write it down, and change nothing until every origin has been evaluated. If you must compare specifications, each one gets its own full rolling origin run and you compare the profiles at the end, accepting that with two test points per specification the comparison is weak. How weak, and what you can honestly say about the difference, is the subject of O27.

The other prohibition is any evaluation scheme that lets a later edition inform a forecast of an earlier one. On a growing show that leaks the trend directly into the fit, and the measured accuracy improves for reasons that have nothing to do with the model. That failure has its own post in O26.

Where this stops

Two test points cannot support a statement about accuracy with an uncertainty attached to it, and no reporting convention changes that. You can compute the mean of two errors. You cannot compute a meaningful standard error of that mean, and quoting one from two observations is arithmetic pretending to be evidence.

The industry's own benchmarking runs on a faster cycle than any single show does. UFI published the 36th edition of its Global Exhibition Barometer in January 2026 from a survey concluded in December 2025, covering 19 markets and regions, and it does that twice a year. Your show produces one final attendance number a year. The mismatch is why lead-time profiles matter so much: they are the only way to generate forecast-and-outcome pairs at a cadence faster than the calendar allows.

There is a limit to that too. Lead-time forecasts of the same edition share an outcome, so a run of four good ones can come from one easy year. The profile tells you how error behaves across the window. It does not tell you how error behaves across years, and only editions can tell you that.

Take your last five editions and the daily registration extract you already keep, and build the eight-cell grid: two held-out editions by four lead times. It is an afternoon of work in a spreadsheet, and the resulting profile will settle more arguments about forecast credibility than another month of model development. The rest of the forecasting methods work should be judged against it.