Post event survey send timing and the trade off between recall and replies

The show closed at four on Thursday. The audience marketing lead wants the survey out before people leave the venue. The research lead wants to wait until Monday so attendees have had time to think about whether the week was worth it. Both have run shows and both are right about something.

Post event survey send timing is one of the few decisions in the whole reporting cycle where the two considerations pull in genuinely opposite directions, and where an organiser can settle the argument with data from their own audience in a single edition.

The two things moving in opposite directions

Replies decay with delay. The invitation competes with whatever has piled up in the attendee's inbox since they left, and by day five the show has become last week.

Recall decays too, and it does not decay evenly. Sudman and Bradburn published a model of the effect in 1973, in the Journal of the American Statistical Association, combining forgetting with telescoping, where a person places an event as having happened more recently than it did. Their finding on remedies is the useful part for a survey designer: records reduce telescoping without reducing omissions, while aided recall reduces omissions and may increase telescoping. There is no single fix that repairs both failures.

Against that, some of what you want to ask does not exist yet on day one. Whether a conversation on the stand turned into a quotation, whether the attendee's team acted on what they saw, whether they would come back given what the week produced. Those answers form over days or weeks, and asking on Friday morning gets you a guess.

So the trade is between a question the attendee can answer accurately and a question worth asking.

It helps to separate the instrument into the two kinds of question before arguing about a date, because they have different expiry times. Ratings of things the person experienced directly, such as queueing at registration, the temperature in hall two, whether the sessions started on time, decay fast and are worth almost nothing by the following week. Judgements about value, such as whether the trip was worth the flight, form slowly and are close to meaningless on the Friday morning while the person is still in a taxi. Put both kinds on one send and you have picked a date that is wrong for half of your questions by construction.

Does the day of the week matter?

Less than the folklore says, and this is worth knowing because it is where teams spend their argument.

Howard, Greaves, Osborne and Sibley randomly assigned 26,126 existing panel participants to receive a survey invitation on each of 14 consecutive days and published the result in the Journal of Survey Statistics and Methodology in 2023. Response rates did not differ significantly by the day of the week the initial email went out. The overall response within 31 days came in at 28.64 per cent. The one exception they found sat in immediate responses, which were lower for Friday and Saturday invitations.

A panel of people who have already agreed to take surveys is a different animal from a trade show attendee list, so the finding does not transfer wholesale. What it does establish is that day-of-week effects are small enough to be hard to detect in a randomised design with 26,126 people. If a real effect is that small, the two hours your team spends debating Tuesday against Wednesday is time taken from the decision that does move the number, which is how many days you wait.

Running the split send

Take one edition and split the verified attendee file at random into two halves. Same instrument, same subject line, same sender. One half gets the invitation at 24 hours after close, the other at day five.

On a show with 18,600 verified attendees the halves are 9,300 each. Say the day-one arm returns 1,023 completed responses, which is 11.0 per cent, and the day-five arm returns 651, which is 7.0 per cent. Mean satisfaction comes in at 7.86 for the early arm and 7.56 for the late one.

Four points of response rate and three tenths of a point on the score. The response difference is worth 372 responses, which on most shows is the difference between reporting a country breakdown and not reporting one.

Randomising the split is what makes the rest of the analysis possible. The two invited populations are identical by construction, so any difference between the two respondent groups was produced by the send date. Without randomisation you have two samples and no way to tell whether the timing did anything.

Three details are worth getting right before you run it. Randomise on the person rather than on the badge type or the country, since splitting on any attribute reintroduces the composition difference you are trying to measure. Hold the reminder schedule identical in both arms, counted from each arm's own send date rather than from a fixed calendar day, or the reminder becomes a second variable. And close both arms on the same number of elapsed days, because a day-one arm left open for three weeks against a day-five arm closed after ten days is comparing collection windows as much as send dates.

The cost of running it is one extra segment in the email platform. The cost of not running it is that the send date gets set by whoever is loudest in the debrief and then persists for a decade, which is roughly how most organisers arrived at the date they currently use.

Is the gap composition or recall?

Three tenths of a point could mean two quite different things, and the difference decides what you do about it.

Start with composition. Returning attendees make up 54 per cent of respondents in the day-one arm and 63 per cent in the day-five arm, so the later send lost first-time attendees disproportionately. Suppose the pooled data puts returning attendees at 7.4 and first-timers at 8.4, which happens at plenty of shows where novelty flatters the rating.

The day-one arm's expected mean from that mix is 0.54 times 7.4 plus 0.46 times 8.4, or 7.86. The day-five arm's is 0.63 times 7.4 plus 0.37 times 8.4, or 7.77. So composition alone accounts for 0.09 of the movement.

The observed movement was 0.30. That leaves 0.21 unexplained by who replied, which means the same kind of person rated the show about two tenths lower five days later. Check it directly by comparing returning respondents in one arm against returning respondents in the other, and if both subgroups are down by roughly 0.21, you have measured a genuine timing effect on the rating rather than a sampling artefact.

Which of the two you are looking at changes the remedy. A composition effect is repairable after the fact by weighting, and the arithmetic for that is weighting survey results in D26. A recall effect is a property of the instrument and the only remedy is to send earlier.

What I would actually do

Send the experience questions at 24 hours and accept that some of what you want to know is out of reach at that point.

Then send a second instrument, three questions long, at three to four weeks, to the people who completed the first one. Ask whether anything came of the show: a supplier contacted, a purchase made, a project started. Those three questions have a low completion rate and it does not matter, because the population is already characterised by the first survey and you are measuring an outcome rather than a rating.

Two sends beat one compromise send at day three, which gets a mediocre response rate on questions people cannot yet answer. The attendee analytics reporting then carries two dated figures with different properties instead of one figure that is slightly wrong about both things.

One practical caution on the second send. Suppress anyone who did not complete the first, because chasing non respondents at week four collects a small, strange sample and attaches it to the same series.

Where this stops

A split send measures the effect of timing on your audience this year. It does not transfer cleanly to your other shows, and it may not transfer to next year's edition of the same show if the mix changes.

The effect also interacts with things the test holds constant. Instrument length changes who finishes, and a long survey at day five may perform quite differently from a short one, which is survey length and completion rate in D24. Both arms of the split still exclude everyone who never opens the invitation at all, and sizing that group against the full file is post show survey response bias in D23.

There is a boundary with the exhibitor side that matters operationally. Exhibitor feedback has a hard deadline that visitor feedback does not, because the results pack has to be in the exhibitor's hands before the renewal conversation happens, and that constraint outranks recall entirely. When that pack lands is when to send exhibitor results in D36, and it should be decided from the sales calendar.

Run the split on your next edition. It costs one extra send in the platform and an hour setting up the randomisation, and it replaces a recurring argument with two numbers you can point at. If the response gap turns out to be under a point and the score gap under a tenth, you can stop having the argument entirely and send whenever suits the team.