Coding open text survey comments from a show into something teams can fix
Code open text survey comments against a fixed frame of twelve to fifteen operational categories chosen so each one names a team that could fix it. Double code a ten per cent sample to measure agreement, and sub-code any category holding more than about a quarter of the comments, because one large category usually hides three different problems.
Eighteen hundred and forty people typed something into the box at the end of the survey. The report contains a word cloud, and the largest word in it is show.
Coding open text survey comments is the least glamorous job in the whole post-show cycle and the one with the highest return, because the free text is where the specific, actionable complaint lives. Somebody wrote that the badge reprint desk had one person on it at nine on the Tuesday. That sentence is worth more than the satisfaction mean, and the word cloud has turned it into the word badge in eleven point type.
What does a word cloud do to 1,840 comments?
Three things, all of them bad.
It splits a single complaint across tokens. Queue, queuing, queued, line, waiting and wait are six words describing one problem, and the cloud ranks each of them separately, so the biggest issue at the show can appear as six medium-sized words while a word that happens to have one spelling dominates the picture.
It discards polarity. The wifi was excellent and the wifi was unusable both contribute the word wifi. A cloud where wifi is large tells you people mentioned it and nothing about whether they were pleased.
It has no denominator. A prominent word could have come from 40 people or 400 and the visual gives no way to tell, which means nobody can decide whether it justifies spending money.
Building a code frame that names an owner
The frame is a fixed list of categories written before coding starts, and the test for each category is whether you can name the team that would fix it.
Registration queueing has an owner. Attendee experience does not. Catering price has an owner. Value for money does not. That test alone removes most of the categories that get proposed in the first meeting, and it is worth being blunt about it, because a frame full of themes produces a chart nobody can act on and a frame full of operational categories produces a list of jobs.
Twelve to fifteen categories is the working range for a general post show survey. On one show's comments the frame ran to twelve: registration queueing, badge collection and reprints, catering quality, catering price, wayfinding and signage, seating and rest areas, wireless connectivity, session content quality, floor plan and layout, staff helpfulness, hall temperature, and the event app. Plus an unclassified bucket, which exists so coders stop forcing awkward comments into the nearest category.
Watch the unclassified bucket. Under about five per cent it is doing its job. Over fifteen and your frame is missing a category that this year's show produced, which is a finding in itself.
Give each category a written inclusion rule and one real example comment. Two coders working from a category name alone will diverge within twenty comments, and every hour spent writing rules saves three in reconciliation.
Allow multiple codes per comment, with a cap of two. Real comments carry two complaints often enough that forcing a single code throws away a fifth of the content, and a cap of two stops a coder from tagging a long comment with six categories to be safe. Record polarity as a separate flag on each code rather than as a category of its own, because a positive mention of catering and a negative one belong in the same operational row and get counted differently within it. That flag is what lets you say 331 comments mentioned catering quality, of which 268 were complaints, which is a sentence an operations lead can use.
How do you know the coding is reliable?
Double code a sample and measure agreement beyond chance. Ten per cent is the usual share and it is enough.
Take the 1,840 comments. A ten per cent sample is 184, coded independently by two people who have not discussed the individual comments. Suppose they assign the same category to 152 of them, which is 82.6 per cent raw agreement.
Raw agreement flatters, because two coders assigning categories at random would agree some of the time, and the more skewed the distribution the more often that happens. Chance agreement is the sum of the squared category shares. With queueing at 31 per cent, catering quality at 18, wayfinding at 11, seating at 9, connectivity at 7, session content at 6, catering price at 5, badge collection at 4, layout and staff at 3 each, temperature at 2 and the app at 1, the squares sum to 0.1636.
Working the agreement arithmetic
Kappa is observed agreement minus chance agreement, divided by one minus chance agreement.
That is 0.826 minus 0.1636, over 1 minus 0.1636. The numerator is 0.6624 and the denominator is 0.8364, so kappa is 0.79.
Landis and Koch published the benchmark scale for reading that number in Biometrics in 1977, in their paper on the measurement of observer agreement for categorical data. Their bands run 0.01 to 0.20 as slight, 0.21 to 0.40 as fair, 0.41 to 0.60 as moderate, 0.61 to 0.80 as substantial, and 0.81 to 1.00 as almost perfect. A kappa of 0.79 sits at the top of substantial, which for an operational code frame is a working result.
The reason to compute it at all is that it tells you which fix to apply. Below about 0.6 the problem is almost always the frame rather than the coders, and retraining people to apply an ambiguous category harder makes agreement worse. Between 0.6 and 0.8 the usual culprit is two or three specific category pairs, and a confusion matrix on the 32 disagreements will name them in five minutes. Above 0.8 you can code the remaining 1,656 comments single-handed.
Kappa and alpha measure related things on different scales
There is a trap here worth naming, because it produces confident wrong readings.
Krippendorff's alpha is a different statistic from Cohen's kappa, and it comes with its own published thresholds. Krippendorff's 2004 treatment in Content Analysis: An Introduction to Its Methodology sets the customary requirement at alpha of 0.800 or above, with 0.667 as the lowest limit at which tentative conclusions remain acceptable and anything below that discarded.
A kappa of 0.79 is not an alpha of 0.79, and reading it against Krippendorff's 0.800 line to conclude that your coding falls just short of publishable is a category error. Pick one statistic, state which one you used, and read it against that statistic's own benchmarks. Alpha is the better choice when you have more than two coders or missing codes, and kappa is fine for the two-coder, complete-data case that most show teams actually run.
Sub-coding the big category
Queueing took 31 per cent of the comments, which is 570 of the 1,840. That is the largest number in the analysis and on its own it is useless, because it names four different problems with four different owners.
Sub-code it. On this show the 570 split into 312 about the entry queue before half past nine, 141 about the badge reprint desk, and 117 about the cloakroom. Registration operations owns the first two, the venue owns the third, and the fixes are a staffing change, a second reprint station, and a conversation with the hall.
Any category above about a quarter of the comments deserves the same treatment. Below that, the sub-codes will be thin enough that you are reading individual comments anyway, which is fine and faster.
Keep the denominators straight while you do it. The 1,840 comments came from 2,240 respondents, so 82.1 per cent of respondents wrote something. The 570 queueing comments are 25.4 per cent of respondents. Against the verified attendance of the show they are a much smaller share still. All three numbers are true and only one of them belongs in the sentence you write, so name which population you are quoting every time. That discipline is the same one the rest of the attendee analytics pack runs on.
Where this stops
Coded comments tell you what the people who wrote something chose to write about. Nobody writes a comment to say that registration worked, so the frame is structurally weighted towards failure, and the absence of praise for a category means nothing at all.
The second limit is that comment volume tracks how annoying a problem was rather than how many people it affected. A ninety second queue that made a thousand people mildly impatient generates fewer comments than a broken lift that stranded forty. Both are worth knowing and the comment counts rank them the wrong way round for a budget decision. Cross-check the big categories against something measured, such as scan timestamps at the entry, before spending.
There is also a limit on the frame itself. It was written for this year's show, and a frame that stays fixed for five editions will stop capturing what the event has become, while a frame that changes every year destroys the series. My compromise is to hold the frame for three editions, then revise it and recode one previous year with the new frame so there is an overlap to read the break against. Whose job each of those fixes becomes, and by when, belongs on the post show close out timeline in D31.
Take last edition's comment export, pick 100 comments at random, and code them yourself against a frame you write in the same hour. You will discover which categories are ambiguous faster than any planning meeting, and the frame that comes out of it is the one to hand to two coders and the post event survey design work in D22, alongside the objective-linked open box that the exhibitor satisfaction survey in D28 puts on the exhibitor side.
Questions people ask about coding open text survey comments
- How many categories should a comment code frame have?
- Twelve to fifteen for a general post show survey. Fewer and the largest category swallows a third of your comments and names no owner. More and the double coded sample stops producing stable agreement, because two coders will separate hairs the categories were never meant to split. Add an unclassified bucket and watch its size.
- How do I check that comment coding is reliable?
- Have two people independently code the same ten per cent sample, then compute agreement beyond chance. Landis and Koch's 1977 benchmark scale in Biometrics reads kappa of 0.41 to 0.60 as moderate, 0.61 to 0.80 as substantial and above 0.81 as almost perfect. Below about 0.6 the code frame needs rewriting rather than the coders retraining.
- Why is a word cloud a bad way to read show feedback?
- It splits one complaint across several tokens, since queue, queuing, line and waiting all count separately, and it discards polarity, so praise and complaint about the same thing merge into one large word. It also has no denominator, which means nobody can tell whether a prominent word came from 40 people or 400.
Related reading
- Post event survey design for a show that already has your data
- Exhibitor satisfaction survey questions that predict whether they come back