Comparing two forecasting methods when you only have four error pairs
Comparing two forecasting methods means testing whether their average loss differential differs from zero. Diebold and Mariano (1995) give the statistic; its normal reference distribution is badly oversized in small samples. Harvey, Leybourne and Newbold (1997) supply a correction and a t reference. On four pairs, report the sign and the size.
Two people have built forecasts for the same show. One uses a pickup model, one uses a pooled regression across the portfolio. Both have been run against four held-out editions and both have four errors. The pickup model looks better on three of the four.
The question in the room is whether that is a real difference or four coin flips. Comparing two forecasting methods properly has a standard answer, and the standard answer behaves badly at this sample size in a way that is worth understanding rather than working around.
Diebold and Mariano (1995), in the Journal of Business and Economic Statistics, proposed tests of the null hypothesis of no difference in the accuracy of two competing forecasts. Their framing is deliberately loose about everything except the comparison: the loss function need not be quadratic and need not even be symmetric, and the forecast errors may be non-Gaussian, non-zero mean, serially correlated and contemporaneously correlated.
What the test is actually comparing
Do not compare two accuracy figures. Compare the paired differences.
For each held-out edition, compute the loss under method A and the loss under method B, using whatever loss you actually care about, and take the difference. Call it d for that edition. You now have a short series of loss differentials, and the null hypothesis is that its mean is zero.
The statistic is the mean of the differentials divided by an estimate of the standard error of that mean. Under the null, and asymptotically, it is standard normal.
The pairing is what makes this work at small sample sizes at all. Both methods faced the same edition with the same shocks, so whatever made 2024 a hard year to forecast affects both losses and largely cancels in the difference. Comparing two independent accuracy averages throws that cancellation away and needs far more data.
Which loss you difference is a real decision and it changes the answer. Absolute error, squared error and scaled error will not rank two methods identically, and picking the one that favours your preferred method after seeing the results is the obvious abuse. Fix the loss before you look, and if the loss is a scaled one, the choice of denominator is O29's subject.
Why is the normal reference distribution wrong at n equals four?
The normal reference is an asymptotic result. It assumes the variance estimate in the denominator is close to the true variance, which requires enough observations to estimate a variance.
With four observations you are estimating a variance from three degrees of freedom, and that estimate is itself extremely noisy. When the estimate comes in low, the statistic comes in large, and the normal reference happily calls it significant. The test rejects far more often than its nominal size says it should.
Harvey, Leybourne and Newbold (1997), in the International Journal of Forecasting, examined the behaviour of the tests and proposed modifications designed to circumvent shortcomings in the original formulations. The modification, as implemented in the forecast package for R, multiplies the statistic by a factor of the square root of n plus one minus twice the horizon plus the horizon times the horizon minus one over n, all divided by n, and compares the corrected value against a t distribution with n minus one degrees of freedom.
At n equal to 4 and a one-step horizon, the factor is the square root of 4 plus 1 minus 2, all over 4, which is the square root of 0.75, which is 0.866. So the statistic shrinks by 13.4 per cent. Then the reference distribution changes from a normal, whose two-sided five per cent critical value is 1.96, to a t with three degrees of freedom, whose two-sided five per cent critical value is 3.182.
Those two changes compound. The threshold on the original statistic moves from 1.96 to 3.182 divided by 0.866, which is 3.675. The bar has almost doubled.
Working four error pairs through the corrected test
Take the two methods from the opening. Score both on absolute percentage error, and let the differentials, method A minus method B, come out as minus 0.9, minus 2.1, plus 0.4 and minus 1.6 percentage points. Negative means method A had the smaller error that year.
The mean is minus 4.2 over 4, which is minus 1.05 percentage points. The sample standard deviation of the four differences is 1.08, so the standard error of the mean is 1.08 divided by the square root of 4, which is 0.54.
The original statistic is minus 1.05 divided by 0.54, which is minus 1.94. Against a standard normal that is a two-sided p-value of 0.053, which is the sort of number that gets rounded down in a meeting and described as significant.
Now apply the correction. Multiply by 0.866 and you get minus 1.68. Against a t distribution with three degrees of freedom the two-sided p-value is 0.19.
The same four numbers. One reading says the difference is on the edge of conventional significance; the other says it is entirely unremarkable. The second reading is the correct one, and the gap between them is the whole reason Harvey, Leybourne and Newbold wrote the paper.
What difference would you have needed to detect?
This is the calculation that settles the argument, and almost nobody runs it.
To reject at the five per cent level with four pairs, the corrected statistic has to exceed 3.182, so the original statistic has to exceed 3.675, so the mean differential has to exceed 3.675 times the standard error of 0.54, which is 1.99 percentage points.
Read that back. With the level of edition-to-edition variability actually present in this data, method A would have to be about two percentage points better on average, in mean absolute percentage error, before four pairs could establish it. Two percentage points is an enormous margin in attendance forecasting. If one of your two candidate methods were that much better, you would not need a test.
The honest output at n equal to 4 is therefore descriptive. Method A had the lower error in three editions of four, and its average advantage was 1.05 percentage points, with the four differences ranging from minus 2.1 to plus 0.4. Every number in that sentence is a fact about the data. None of them is a significance claim, and the absence of one is the finding.
Pooling across the portfolio, and what it costs
The way out is more pairs, and the only place they exist is the rest of the portfolio.
Run both methods across eight shows with two forward origins each and you have sixteen paired differentials. The correction factor becomes the square root of 15 over 16, which is 0.968, close enough to one to stop mattering. The t distribution has fifteen degrees of freedom and a two-sided critical value of 2.131.
With the same standard deviation of 1.08, the standard error of the mean falls to 1.08 over 4, which is 0.27. The detectable mean difference becomes 2.131 divided by 0.968, times 0.27, which is 0.60 percentage points.
Going from four pairs to sixteen took the detectable effect from 1.99 percentage points to 0.60, a factor of more than three. That is the argument for evaluating a forecasting change at portfolio level even when each show is forecast separately, and generating those forward origins is O25's subject.
The cost is an assumption. Diebold and Mariano's variance estimate handles serial correlation in the differentials, and it does not handle the correlation that comes from eight shows sharing a bad year. If 2024 was hard for the whole portfolio in a way that favoured one method, sixteen differentials contain far less independent information than sixteen suggests, and the effective sample size sits somewhere between the number of shows and the number of pairs. Reporting the test alongside a count of how many shows favoured each method is the cheap defence, because a method that wins on seven shows of eight is telling you something the pooled average cannot.
Where this stops
A test with no power tells you nothing when it fails to reject, and the four-pair test has almost no power. Treating a non-rejection as evidence that the two methods are equivalent inverts the logic completely, and it is the most common mistake made with this test in practice.
The correction also fixes the wrong thing for this purpose. Harvey, Leybourne and Newbold repaired the size of the test, meaning how often it falsely rejects. Nothing about the correction improves its ability to find a real difference. A correctly sized test that cannot detect anything smaller than two percentage points is more honest than an oversized one, and it is not more useful.
There is a deeper limit that no sample size fixes. The test compares average loss, and average loss is not what makes a forecasting method worth adopting. A method that is slightly worse on average but never produces a catastrophic miss is often the better operational choice, and that judgement lives in the shape of the error distribution rather than its mean. Before running any of this, it is worth asking whether either method beats simply repeating last year's number, which is the ledger O30 sets out.
Take the two candidate methods you are currently arguing about and write their paired differences in a column. Compute the mean, the standard deviation, and then the detectable difference at your sample size using the arithmetic above. If that threshold is larger than any improvement you could plausibly expect, stop testing and start collecting more pairs, because no amount of statistical care will rescue the evidence you have. The rest of the forecasting methods work is easier once that number is written down.
Questions people ask about comparing two forecasting methods
- What is the Harvey, Leybourne and Newbold correction?
- It multiplies the original statistic by the square root of the quantity n plus one minus twice the horizon plus the horizon times the horizon minus one over n, all divided by n, then compares the result against a Student t distribution with n minus one degrees of freedom rather than a standard normal.
- How many paired forecast errors do you need to detect a difference?
- It depends on how variable the differences are. With four pairs and a standard deviation of about one percentage point, you would need an average advantage near two percentage points before the corrected test rejects at the five per cent level. Sixteen pairs brings that threshold down to about six tenths of a point.
- Does failing to reject mean two methods are equally accurate?
- No. A test with almost no power fails to reject nearly everything, including large real differences. On four pairs a non-rejection is uninformative about whether the methods differ, and reporting it as evidence of equivalence inverts the meaning of the result.
Related reading
- Rolling origin evaluation for a show that produces one number a year
- Mean absolute scaled error compares accuracy across shows of different sizes
- Forecast value added tells you whether the model beat last year