Mean absolute scaled error compares accuracy across shows of different sizes
Mean absolute scaled error divides each absolute forecast error by the in-sample mean absolute error of a naive one-step forecast, so the result carries no units and can be pooled across series. A value below one beats that benchmark. Hyndman and Koehler (2006) proposed it in the International Journal of Forecasting.
The portfolio review has eight accuracy figures on one slide. The flagship posts 4 per cent. A small show in an adjacent vertical posts 9 per cent. The flagship's show director points at the gap and the room accepts it, because 4 is obviously better than 9.
Nobody in the room can say whether the flagship's forecasting is actually better, because the two numbers are measured on different scales of difficulty. Mean absolute scaled error is the standard repair. It divides each error by the error a naive forecast would have made on the same series, which strips out both the size of the show and how hard it is to predict.
Hyndman and Koehler (2006) proposed it in the International Journal of Forecasting for exactly this purpose, arguing that the measures then in use, including those from the M-competitions, are degenerate in commonly occurring situations, and that scaled error should become the standard for comparing accuracy across multiple series.
What does dividing by the naive error buy you?
The construction is one division. Take the absolute error of your forecast. Divide it by the average absolute change between consecutive observations in the training data.
Hyndman and Athanasopoulos (2021) give the scaled error in the third edition of "Forecasting: Principles and Practice" as the error divided by the mean of the absolute differences of consecutive training observations, which they describe as the average one-step naive forecast computed on the training data. The mean absolute scaled error is the average of the absolute scaled errors, and a value below one indicates the forecast beat that benchmark.
Three properties follow, and all three matter for a portfolio.
The measure has no units, so a scaled error from a 43,000 attendee show and one from a 4,600 attendee show are the same kind of quantity and can go in the same average. It is defined when the actual is zero, unlike anything with the actual in the denominator, which is why it survives the situations that break percentage error. And its reference point is a forecast anyone can compute, which means the number carries an interpretation rather than only a ranking.
Two shows, two accuracy figures, and a reversed ranking
Put the flagship and the small show through it.
The flagship's last five editions ran 38,400, 39,900, 41,100, 42,000 and 43,200. The four consecutive absolute differences are 1,500, 1,200, 900 and 1,200. Their mean is 4,800 over 4, which is 1,200.
Its held-out forecast missed by 1,730 people, which against an actual of 43,200 is 4.0 per cent. Scale it: 1,730 divided by 1,200 is 1.44.
The small show's last five editions ran 3,050, 3,700, 3,300, 4,150 and 4,600. The absolute differences are 650, 400, 850 and 450, with a mean of 2,350 over 4, which is 587.5.
Its held-out forecast missed by 414 people, which against an actual of 4,600 is 9.0 per cent. Scale it: 414 divided by 587.5 is 0.70.
The ranking reverses. The flagship, at a MASE of 1.44, did worse than simply writing down last year's number. The small show, at 0.70, cut the naive error by nearly a third. The 4 per cent that carried the room was a statement about how stable the flagship is, and the model contributed nothing to it.
That reversal is not a curiosity. It is the normal outcome when a portfolio has one large steady show and several volatile ones, and it means the show most likely to be praised for its forecast accuracy is often the one where forecasting is adding the least.
Work out what the flagship would have needed to hit parity. A MASE of 1.0 means an absolute error equal to the denominator, which is 1,200 people, or 2.8 per cent of 43,200. So the flagship's model had to land inside 2.8 per cent to be worth running at all, and it came in at 4.0. On the small show, parity means an error of 587.5 against an actual of 4,600, which is 12.8 per cent. Two shows, two break-even accuracy thresholds, and they differ by a factor of more than four. Any single accuracy target applied across a portfolio, whether it is five per cent or ten, is asking eight shows to clear eight different bars while pretending it is one.
The denominator is estimated from four numbers
Here is where annual editions make trouble, and the arithmetic is worth doing because it changes how much precision you should claim.
The small show's denominator came from four absolute differences: 650, 400, 850 and 450. Their mean is 587.5 and their sample standard deviation is 206. The standard error of the mean is 206 divided by the square root of 4, which is 103.
So the denominator is 587.5 give or take about 103, a relative standard error of 17.5 per cent. Push that through. At a denominator of 484.7 the MASE is 0.85. At 690.3 it is 0.60. The point estimate of 0.70 is carrying roughly 0.12 of wobble from the denominator alone, before you account for any uncertainty in the numerator.
A reported MASE of 0.70 on four differences is therefore consistent with anything from about 0.6 to about 0.85, and a difference between two shows of 0.70 and 0.80 is not a difference you can act on. Report scaled error to two significant figures at most, and treat gaps smaller than about 0.15 as noise on this much history.
The fix is more differences, and the only place they exist is either a longer history or the booking window. If you scale against within-window changes rather than edition-to-edition changes, the denominator is estimated from dozens of observations and the wobble collapses. The cost is that you have changed what the benchmark means, and you must then say which naive forecast you scaled against every time you quote the number.
Does a MASE below one mean you beat the naive forecast?
Not quite, and the gap is worth knowing before you defend the number to somebody sharp.
The denominator is computed in sample, on the training data. The numerator is computed out of sample, on the held-out edition. So the comparison is between your out-of-sample error and the naive forecast's in-sample error, and those are not the same standard.
On a series with no trend the difference is small. On a growing show it can matter, because the naive forecast's error out of sample tends to be larger than its error in sample when growth is accelerating, and your MASE therefore flatters the naive benchmark and understates your own advantage.
If you want the clean comparison, compute it directly. Run the naive forecast through the same rolling origin you used for the model, get its out-of-sample absolute errors, and take the ratio of your mean absolute error to its mean absolute error. That is a relative absolute error rather than a scaled error, it answers the question people think MASE answers, and building it into a step-by-step audit of the forecasting process is the ledger O30 sets out.
When the denominator misbehaves
Two situations break it outright and both occur in event portfolios.
A show that has hit exactly the same capped figure for several editions running, because the hall is full and registration closes when it fills, has consecutive differences of zero. The denominator is zero and the scaled error is undefined. That is the measure behaving correctly: a series with no variation has a perfect naive forecast, and there is nothing for a model to improve on.
The opposite case is a show that moved venue or changed format mid-history. One enormous difference in the training data inflates the denominator, every subsequent scaled error looks small, and the model appears to be performing well because the benchmark was made artificially bad. Check the four differences by eye before trusting any MASE computed from five editions. If one of them is three times the others, the number is being set by a single structural event.
Where this stops
Scaled error rewards forecasting a series that moves a lot, and that bias is built into the construction rather than being a flaw in any particular application.
A strongly trending show has large consecutive differences, so its denominator is large and any competent forecast scores well. A flat show has small differences, a small denominator, and a MASE above one from a forecast that is genuinely quite good in absolute terms. Comparing MASE across shows removes the scale effect and leaves the volatility effect, which is better than what percentage error does and is not neutral.
The practical consequence is to compare scaled error across shows for ranking how much value the forecasting process adds, and never to read it as a statement about which show is easier to run. Two shows at 0.9 and 1.1 differ in how much their models help, and that is all the number carries.
Whether a gap of that size is real, given that most portfolios have two or three forward test points per show, is a separate question with an uncomfortable answer in O27, and whether percentage error should have been in the conversation at all is covered in what happens on small counts in O28.
Take the eight accuracy numbers on your current portfolio slide and recompute every one as a scaled error, using each show's own four consecutive edition differences as the denominator. It is one extra column in a spreadsheet you already have. Count how many come out above one, because those are the shows where the forecasting process is currently losing to a copy of last year's figure, and that count is the most useful single output of the whole forecasting methods exercise.
Questions people ask about mean absolute scaled error
- What does a MASE of 1.4 mean?
- It means the forecast error was 1.4 times the average error a naive one-step forecast made on the training data. Anything above one is worse than that benchmark. On annual show editions the naive forecast is simply repeating the previous edition's figure, so a MASE above one says the model added error rather than removing it.
- Why not just use percentage error to compare two shows?
- Because percentage error is scaled by the level of the series and difficulty is not. A large stable flagship can post a low percentage error while performing worse than repeating last year, and a small volatile show can post a high percentage error while comfortably beating that benchmark. Scaling by the naive error compares like with like.
- How is the MASE denominator calculated for annual editions?
- Take every consecutive pair of editions in the training data, compute the absolute difference between them, and average those differences. Five editions give four differences. That average is the in-sample mean absolute error of a naive forecast that predicts each edition using the previous one.
Related reading
- Comparing two forecasting methods when you only have four error pairs
- Why MAPE misleads on small counts such as a forty person session
- Forecast value added tells you whether the model beat last year