Asymmetric forecast loss when running out costs more than overbuilding

The shuttle contract has to be signed nine weeks out. Coaches are booked in whole units, the operator wants a number, and everyone in the room knows that turning attendees away at a hotel pickup point on day one is a different kind of bad from running two coaches half empty.

Asymmetric forecast loss is the framework for that decision, and the useful part of it is a single division you can do on the back of the contract. The number you should commit against is a quantile of the demand distribution, the level of that quantile is fixed by the ratio of the two unit costs, and it is almost never the median that your forecasting process has been quietly producing.

Gneiting (2011), in "Quantiles as optimal point forecasts" in the International Journal of Forecasting, puts the general result plainly: quantiles arise as optimal point forecasts under a class of loss functions that nests the asymmetric piecewise linear loss, and the level of the quantile depends on an asymmetry parameter reflecting the possibly distinct costs of underprediction and overprediction.

Why does the optimal number sit above the median?

Write the two costs per unit. Call the cost of one unit of unused capacity co, and the extra cost of one unit of demand you failed to serve cu.

Commit to a level f against a random demand Y. The expected cost is co times the expected surplus plus cu times the expected shortage. Differentiate with respect to f. The surplus term contributes co times the probability that demand falls below f, which is co times F(f). The shortage term contributes minus cu times the probability that demand exceeds f, which is minus cu times one minus F(f).

Set the derivative to zero. You get co times F(f) equal to cu times one minus F(f), which rearranges to F(f) equal to cu divided by cu plus co.

That fraction is the critical ratio, and it is the quantile level you should be forecasting. When the two costs are equal it is 0.5 and the median is right. When a shortage costs three times a surplus it is 3 over 4, which is 0.75. When a shortage costs nine times a surplus it is 0.9.

Nothing in that derivation assumed a distribution. It assumed the two costs are linear per unit and that you know them, which are the assumptions worth arguing about, and the argument is at the end of this post.

The critical ratio worked on shuttle coaches

Put the shuttle contract through it.

A 50-seat coach costs 780 pounds for the day whether it runs full or empty, so the cost of one unused seat is 780 divided by 50, which is 15.60 pounds. An attendee left at the pickup point takes a taxi and claims it back at an average of 62 pounds. The extra cost of being one seat short is therefore 62 minus the 15.60 you did not spend on the seat, which is 46.40 pounds.

The critical ratio is 46.40 divided by 46.40 plus 15.60. The denominator is 62.00, so the ratio is 0.748.

Round it and you are committing at the 0.75 quantile of shuttle demand. Suppose your demand model puts the median at 3,150 passengers and the 0.75 quantile at 3,480. Divide each by 50 seats. The median implies 63 coaches. The 0.75 quantile implies 69.6, which rounds up to 70.

Seven extra coaches at 780 pounds is 5,460 pounds of additional committed spend. The question the finance lead will ask is whether that is worth it, and the derivation above is the answer: at these two unit costs, the expected cost of the 63-coach plan is higher than the expected cost of the 70-coach plan, because the 350 passengers sitting between the median and the 0.75 quantile are each worth 46.40 pounds of avoided taxi claim against 15.60 pounds of wasted seat.

Notice how much of the work the taxi reimbursement rate is doing. Reduce it to 30 pounds and the shortage cost becomes 30 minus 15.60, which is 14.40. The ratio is then 14.40 divided by 30.00, which is 0.48, and you should commit slightly below the median. The direction of the asymmetry is a property of your contract terms, and the same show can flip from over-committing to under-committing on a single renegotiated line.

What is your error measure actually asking for?

Here is where most forecasting processes quietly contradict themselves.

Gneiting (2011), in "Making and Evaluating Point Forecasts" in the Journal of the American Statistical Association, argues that comparing point forecasts by an error measure "can lead to grossly misguided inferences, unless the scoring function and the forecasting task are carefully matched." His framework is consistency: a scoring function is consistent for a functional when the expected score is minimised by reporting that functional, and a functional is elicitable when some scoring function is strictly consistent for it.

The consequences are concrete. Squared error is consistent for the mean. Absolute error is consistent for the median, which is why Gneiting notes that under symmetric piecewise linear loss any median is an optimal point forecast. Asymmetric piecewise linear loss at level tau is consistent for the tau quantile.

So if the operations team needs the 0.75 quantile of shuttle demand, and the forecasting team is scored monthly on mean absolute error, the scoreboard is paying for the median. A forecaster who correctly reports the 0.75 quantile will lose on that scoreboard to one who reports the median, every single period, and the incentive works exactly as designed. The problem is that the design was never checked against the decision.

The repair is one line in the reporting pack. Score the number that gets committed against with the quantile score at the same tau you derived from the costs, which Hyndman and Athanasopoulos (2021) set out in the third edition of "Forecasting: Principles and Practice" and note is sometimes called the pinball loss. The estimation side of producing that number is fitting the quantile directly and belongs to O23.

When the same show needs two different taus

An event does not have one critical ratio. It has one per commitment, and they disagree.

Shuttle capacity at 0.75. Catering, where a committed cover costs 18 pounds and an uncatered attendee buys their own lunch at no cost to you, has a shortage cost close to zero and a ratio well below 0.5, so you should commit under the median. Badge stock, where blanks cost pennies and running out stops the registration desk, has a ratio close to 1. Hall hire, where the surplus cost is enormous and the shortage is handled by a waitlist, sits low.

The single point forecast that everyone quotes is therefore wrong for almost every decision it is used for, and the mistake is invisible because the forecast is right in the sense of being the middle of the distribution.

What actually works is to publish the distribution once and let each commitment pick its own level from it. One table of demand quantiles, one column per level, and a note next to each operational commitment saying which level it draws from and why. The critical ratio calculation for each line takes about five minutes with the contract in front of you, and it converts an argument about whether the forecast is optimistic into an argument about unit costs, which is the argument that can be settled.

Where this stops

Two assumptions carry the whole derivation and both are shakier than the arithmetic suggests.

The first is linearity. The formula assumes the tenth stranded attendee costs the same as the first. On a shuttle route that is roughly true up to the point where the queue becomes visible and somebody posts a photograph of it, after which the marginal cost of the next stranded attendee is a reputational cost nobody has a per-unit price for. Once the cost curve bends, the critical ratio is a lower bound on the level you should target, and how far above it you should go is a judgement rather than a division.

The second is that you know cu and co. The surplus cost is usually a contract line and easy. The shortage cost almost never is, because the visible part, the taxi claim, is the part that gets reimbursed, and the invisible part, the exhibitor whose buyer never arrived, does not appear in any ledger. Teams that estimate cu from reimbursements alone will systematically under-provide, and the bias is in the direction that looks fiscally responsible in the quarter it happens.

There is a third limit worth naming. Everything above assumes you have a credible demand distribution to take a quantile from. On a show with five editions of history that distribution is thin, and the 0.75 quantile of a thin distribution inherits every weakness of the estimate. Choosing the right target does not rescue a bad forecast; it stops a good one from being used incorrectly.

Take the next commitment your show has to make and write down the two unit costs from the contract. Divide the shortage cost by the sum of the two. If the answer is not close to 0.5, the number you have been committing against has been the wrong one, and the size of the gap between that quantile and the median is the money the arithmetic just found. How you then score the result matters too, and the choice of measure is covered in what percentage error does on small counts in O28 and scaling error across shows in O29, both of which sit inside the same body of forecasting methods.