Putting an NPS confidence interval on your show score before the board sees it
An NPS confidence interval comes from the variance of promoter share minus detractor share, which is that sum minus the squared score, divided by the response count. A score of 22 on 400 responses carries a standard error near 3.8 points and a 95 per cent band of roughly 14.6 to 29.4.
The score is 22 this year and 19 last year. Somebody has already written the sentence about a three point improvement, and somebody else has already attached it to the operational changes made in registration.
Four hundred people answered the question. An NPS confidence interval on that base is wide enough to contain both years comfortably, and the three point improvement is a statement about which four hundred people happened to reply.
Where the uncertainty in an NPS comes from
The score is a summary of a sample. Run the same survey on a different four hundred attendees from the same audience and you get a different number, and the spread of those hypothetical numbers is what the interval describes.
That much is true of any survey statistic. What makes NPS awkward is that it is a difference of two proportions computed on the same respondents, so the two components are negatively correlated, and the usual proportion interval that survey tools produce does not apply.
The clean way to see it is to stop treating the score as a difference and start treating it as a mean. Give every respondent a value: plus one for a promoter, zero for a passive, minus one for a detractor. The mean of those values is the net promoter score divided by 100, and now you are dealing with an ordinary sample mean whose interval you already know how to compute.
The variance of a difference of two proportions
Work it through once. Write p for the promoter share and d for the detractor share.
The expected value of a single respondent's score is p times one, plus the passive share times zero, plus d times minus one, which is p minus d. That is the NPS in decimal form, as expected.
The expected value of the square of a single respondent's score is p times one, plus d times one, since squaring turns minus one into one, and the passives still contribute nothing. That is p plus d.
Variance is the second moment minus the square of the first, so the variance of one respondent's score is p plus d, minus the quantity p minus d squared. Divide by the number of respondents for the variance of the mean, take the square root, and you have the standard error.
Two things fall out of that formula immediately. Passives contribute to the score not at all and to the variance not at all, so a survey where half the respondents are passives behaves like a survey with half as many people. And the variance depends on the shape of the distribution rather than only on the headline number, which means two shows reporting the same score can carry different bands.
What band does a score of 22 on 400 responses carry?
Take 400 responses split 168 promoters at 42.0 per cent, 152 passives at 38.0 per cent and 80 detractors at 20.0 per cent. The score is 42 minus 20, which is 22.
The variance of one respondent's score is 0.42 plus 0.20, minus 0.22 squared. That is 0.62 minus 0.0484, or 0.5716.
Divide by 400 and you get 0.001429. The square root is 0.0378, which in NPS points is 3.78.
A 95 per cent interval is 1.96 standard errors either side, so 1.96 times 3.78 is 7.4. The band runs from 14.6 to 29.4.
Last year's 19 sits inside that band with room to spare. The three point improvement is a number the report should carry, and the sentence explaining what caused it should not exist.
Compare two independent years directly and the position is worse. The standard error of the difference between two scores each measured on 400 responses is the square root of 3.78 squared plus 3.78 squared, which is 5.35 points. A 95 per cent band on the year-on-year movement is 1.96 times that, or 10.5 points. Movements under about ten points on this base are not distinguishable from resampling the same audience.
Two shows, same score, different bands
Both shows report 22 on 400 responses. The first has the distribution above: 42 per cent promoters, 38 per cent passives, 20 per cent detractors, standard error 3.78.
The second is polarising: 55 per cent promoters, 12 per cent passives, 33 per cent detractors. It also scores 22. Its variance is 0.55 plus 0.33, minus 0.0484, or 0.8316. Divided by 400 that is 0.002079, and the standard error is 0.0456, or 4.56 points.
The polarised show carries a band about 21 per cent wider on the same score and the same response count, because it has fewer passives sitting harmlessly in the middle. Anyone comparing the two shows on the number alone will treat them as identical, and their scores are not equally reliable. Reporting the three shares alongside the score, which net promoter score for events in D27 argues for on separate grounds, also lets a reader see this.
How many responses do you need for a three point band?
Invert the arithmetic. A 95 per cent half-width of three points means 1.96 times the standard error equals 0.03 in decimal terms, so the standard error is 0.015306 and the variance is 0.00023428.
With a variance per respondent of 0.5716, the required count is 0.5716 divided by 0.00023428, which is 2,440.
That is a sobering figure for a single show. Most exhibition surveys land somewhere between 400 and 1,000 responses, which buys a band of plus or minus 7.4 down to plus or minus 4.7. Getting to a three point band means multiplying your response count by six, and the honest options are a longer field, a shorter instrument, or pooling editions.
Pooling is the one worth considering. Two editions of the same show at 400 responses each give you 800 for a two-year average, and the two-year average moves more slowly and means more. It also removes the annual ritual of explaining a movement that did not happen.
The same arithmetic sets a floor on how far you can slice the score. A country breakdown that puts 60 responses in a cell gives that cell a standard error of the square root of 0.5716 over 60, which is 0.0976, or 9.8 points, and a 95 per cent band of plus or minus 19. Reporting a regional NPS on 60 people is reporting the draw rather than the region. Set a minimum cell size before the breakdown is built, publish the cells that clear it, and suppress the rest with a note saying why.
Better intervals than the plain one
The interval above is the plain Wald form, and it degrades at the extremes. A show scoring plus 80 or minus 60 on a small base produces a band that runs off the end of the scale, which is obviously wrong since the score cannot exceed 100.
Rocks addressed exactly this in The American Statistician in 2016, adapting standard interval estimation techniques to the net promoter score and testing their coverage against the largest available database of company scores. The methods that came out best were variations on the adjusted Wald and an iterative score test. The adjusted Wald form adds a small number of pseudo-observations to the counts before computing the interval, which pulls the estimate away from the boundary and gives better coverage at the extremes.
For a score in the middle of the range on several hundred responses, the plain form is close enough that the difference will not change a decision. For a portfolio roll-up where some shows sit at minus 40 and some at plus 55, use the adjusted form, because that is where the plain interval misbehaves.
Weighting complicates it further. If you have post-stratified the responses to match verified attendance, the effective sample size is smaller than the raw count and the band widens accordingly, which is the design effect calculation in weighting survey results in D26.
Where this stops
A confidence interval describes sampling variation and nothing else. It assumes the 400 respondents are a random draw from the audience, which they are not.
If your respondents skew towards returning attendees, the interval around your biased estimate is a precise statement about a wrong centre. The band tells you how much the number would move if you drew a different sample from the same self-selected pool, and self-selection is the larger error at most shows. That is why the interval belongs next to the composition check rather than instead of it, and both belong in the same attendee analytics methodology note.
There is also a limit on what an interval does to a room. A board given a range will often read the bottom of it as the real number, and a report full of bands starts to read as a report with no findings. Presenting an interval so it lands as discipline rather than hedging is a writing problem with real solutions, and it sits with stating uncertainty in show reporting in D40.
One more caution about the measure itself. Keiningham, Cooil, Andreassen and Aksoy examined the net promoter claim in the Journal of Marketing in 2007 using longitudinal data from 21 firms and more than 15,500 interviews in the Norwegian Customer Satisfaction Barometer, and found it no better than other loyalty measures at predicting growth. A tighter interval on a weak predictor is still a weak predictor.
Take your last score and your response count, put them into the variance formula, and write the resulting band on the same slide. If the band contains last year's number, delete the explanatory sentence before anyone reads it.
Questions people ask about nps confidence interval
- How do you calculate a confidence interval for a net promoter score?
- Score each respondent as plus one for a promoter, zero for a passive and minus one for a detractor. The variance of that score is the promoter share plus the detractor share, minus the square of the net score. Divide by the response count, take the square root for the standard error, and multiply by 1.96 for a 95 per cent band.
- How many survey responses does a three point NPS band need?
- About 2,440, for a score of 22 with 42 per cent promoters and 20 per cent detractors. The requirement scales with the variance, so a polarised audience with few passives needs more responses than a moderate one at the same headline score. Most single show surveys land nearer 400 to 1,000.
- Is a three point year on year move in event NPS meaningful?
- Rarely. Comparing two independent samples of 400 responses each, the standard error of the difference is about 5.4 points, so movements smaller than roughly 10 points fall inside sampling noise. Publishing a three point change as a finding trains a board to react to something that has not happened.
Related reading
- Weighting survey results so show feedback matches who actually attended
- Net promoter score for events and what one point of movement means