Running a holdout test for editorial content when attribution will not settle it
A holdout test for editorial content suppresses a content series from a randomly chosen share of the prospect list and compares registration rates between the held-out and exposed groups. The comparison is causal because assignment was random. Size the holdout from the smallest lift worth acting on before running it.
The argument in the room is always the same one. Marketing says the exhibitor economics series drove 380 registrations. Finance says those people were already on the list and would have registered anyway. Both sides have a number, both numbers come from the same warehouse, and there is no version of the attribution model that ends the meeting, which is what a holdout test for editorial content exists to fix.
A holdout is the only instrument here that produces an answer neither side can rebuild in a spreadsheet, because it manufactures the missing comparison instead of estimating it.
Why won't attribution settle this?
Attribution distributes credit among people who registered. Every model in common use, first touch, last touch, linear, time decay or a fitted data-driven variant, is a rule for splitting a conversion that already happened. None of them observes the world where the content did not exist, so none of them can say what would have happened in it.
The best evidence on how far that gap runs comes from Gordon, Zettelmeyer, Bhargava and Chapsky, writing in Marketing Science in 2019. They took fifteen advertising experiments at Facebook, covering about 500 million user-experiment observations and 1.6 billion impressions, and compared the experimental estimates against what standard observational approaches would have reported on the same data. The observational methods often failed to recover the experimental answer, and the direction of the error was not consistent enough to correct for.
That result is about paid advertising and my subject is editorial, so treat it as a caution instead of a measurement of your own programme. The mechanism transfers cleanly enough. People who read your exhibitor economics series are people who were already interested in exhibiting, and no amount of modelling separates the reading from the interest when both are observed together.
What suppression actually buys
Randomly assign every person on the prospect list to one of two groups. The exposed group receives the content series through the channels you control: newsletter, nurture email, retargeting audiences, on-site promotion. The held-out group receives everything else you normally send and nothing from that series.
Registration rate is then measured in both groups over the same window. The difference is caused by the suppression, because nothing else differs systematically between two groups drawn at random from the same list.
The claim you can make afterwards is narrow and worth stating precisely. You have measured the effect of promoting that series to that list over that window. You have not measured the effect of the series existing, because the held-out group can still find the articles through search, and you have not measured the effect of any individual article.
Sizing the holdout before you run it
Do this arithmetic first. It takes five minutes and it is the difference between a test that answers something and a test that returns a shrug.
Start with the base registration rate on your prospect list. Say it is 6 per cent. On 10,000 prospects with a twenty per cent holdout, you have 2,000 held out and 8,000 exposed, expecting about 120 and 480 registrations respectively if the content does nothing.
The standard error of the difference between two proportions is the square root of p times one minus p times the sum of the reciprocals of the two group sizes. Here p times one minus p is 0.06 times 0.94, which is 0.0564. One over 2,000 plus one over 8,000 is 0.000625. Multiply, and you get 0.00003525. The square root of that is 0.00594, so the standard error is 0.59 percentage points.
The smallest difference you can reliably detect is about 2.8 times that standard error, using the conventional five per cent significance level and eighty per cent power. That gives 1.66 percentage points. In registrations, 1.66 per cent of 8,000 is 133.
So the test as specified can find a lift of 133 registrations and will miss a lift of 80. Whether that is good enough depends entirely on what you would do differently at each figure, which is a question to settle before the test rather than after it.
Is twenty per cent the right holdout?
Not usually, and this is where I part company with the standard advice.
Rerun the arithmetic on a fifty fifty split of the same 10,000. One over 5,000 plus one over 5,000 is 0.0004. Times 0.0564 gives 0.00002256, whose square root is 0.00475. The detectable difference falls to 1.33 percentage points, from 1.66. The same list, the same spend, and the test got roughly twenty per cent sharper purely from how the people were divided.
The cost of a fifty fifty split is that half your list stops receiving a series you believe in, which is a real commercial cost for a fixed period. The cost of a twenty per cent split is that you may run a three month test and learn nothing. Teams reliably underweight the second cost because it arrives later and has no owner.
If you want to detect 1.5 points at an eighty twenty split, solve the same expression backwards. You need a standard error of 1.5 divided by 2.8, which is 0.536 percentage points, so a variance of 0.00002871. With 0.0564 on top and the reciprocals summing to 6.25 divided by the list size, the list size comes out at about 12,300. That is a quarter more prospects for a fifth less sensitivity than the balanced split on 10,000 would have given you.
My rule is to run balanced until the effect is established, then move to a small permanent holdout for monitoring once you know roughly how big the effect is.
Four ways the test leaks
Every holdout on an owned list leaks. The question is whether it leaks enough to matter, and all four of these are measurable before you start.
- Organic discovery. The suppressed group can still search for the article and find it. Measure the share of your content sessions that arrive from organic search. If it is 70 per cent, your suppression is only touching the promotional channel and you should say so in the write-up.
- Forwarding and sharing. One exhibitor sends the newsletter to three colleagues, two of whom are in the holdout. Small on most lists, large on lists dominated by a handful of big accounts. Randomise by account instead of by person where a few companies supply most of your registrations.
- Sales contact. A salesperson pastes the article into an email to a held-out prospect. This is the leak that actually breaks tests, because it is correlated with account value. Tell the sales team the test exists, or accept the contamination and report it.
- Cross-brand exposure. A portfolio that publishes the same material on a sibling site has effectively exposed everybody. Suppress across the portfolio or pick a series that only one brand runs.
Count the leaks and report them as a share of the holdout. A test where 4 per cent of the held-out group saw the content anyway is fine and the bias is towards understating the effect. A test where 30 per cent did is not a test.
When the list is too small to test
Some brands cannot get there, and the honest response is to say so early.
Lewis and Rao, in the Quarterly Journal of Economics in 2015, worked through twenty-five large field experiments representing $2.8 million of advertising spend and found confidence intervals on return that routinely exceeded 100 percentage points. Their explanation is that individual-level sales are extremely volatile relative to the per capita cost of the advertising, with a coefficient of variation around 10 being common, which is why informative experiments need enormous samples.
Registration is a cleaner outcome than sales revenue, since a registration is a zero or a one and there is no long tail of order values, so event brands are in a better position than the advertisers in that paper. The lesson still holds at the bottom of the range. A brand with 3,000 prospects and a 6 per cent base rate cannot detect anything smaller than about three percentage points, and a three point effect from one content series would be remarkable.
Two options remain at that size. Pool across editions, running the same suppression for two or three cycles and analysing them together, which is slow and vulnerable to anything else changing. Or accept a descriptive measure instead, reported honestly as such, which is where counting assisted registrations as coverage belongs.
Where this stops
A holdout answers one question about one series over one window. It says nothing about the piece you published last Tuesday, and it cannot be run per article without splitting the list into fragments too small to measure.
The test also inherits its runtime from something outside it. If your registrations arrive a median of six weeks after first contact with a piece, a four week holdout reads mostly noise, so the lag distribution has to be measured before the test is scheduled. Run the holdout for at least the ninetieth percentile of that lag.
There is a growing hole underneath the whole method, which is that exposure is getting harder to observe. A prospect who encounters your brand inside a generated answer, reads a summary of your article and never lands on the page is exposed by any sensible definition and sits in your held-out group as untouched. Google's own guidance asks publishers whether content is "primarily made to attract visits from search engines" (Google Search Central, 2025), and the honest answer for a lot of event editorial is that visits were always a proxy for reach. Counting appearances inside answer engines is a separate measurement and it does not fold into this one.
Take your prospect list count and your base registration rate, and do the standard error arithmetic above on a balanced split. If the detectable lift comes out larger than any effect you would believe, you have learned something useful before spending anything, and whoever owns the editorial operation needs that number before they promise a causal answer.
Questions people ask about holdout test for editorial content
- Why not just use attribution instead of a holdout test?
- Attribution assigns credit among people who converted. It cannot tell you what those people would have done with no content at all, because it never observes that world. Gordon and colleagues compared observational models against fifteen randomised experiments at Facebook in 2019 and found the observational estimates frequently missed the experimental ones, sometimes by a wide margin.
- How large does the prospect list need to be for a holdout test?
- It depends on the lift you want to detect. At a six per cent base registration rate, a list of 10,000 split eighty twenty detects roughly a 1.7 percentage point difference with conventional power. Detecting 1.5 points at that split needs about 12,300 prospects. Halving the target lift roughly quadruples the list you need.
- Is it acceptable to withhold content from part of your audience?
- Suppression means withholding promotion of a series from a randomly chosen group for a fixed period, and the content stays publicly available to anyone who finds it. Keep the period short, exclude anyone with a service or safety reason to receive the material, and write down the start date, the end date and who approved it before the test runs.
Related reading
- Measuring content to registration lag so editors stop judging pieces too early
- Counting content assisted registrations without pretending the content caused them