Computer vision people counting on a show floor and its failure modes

The screen in the show office says 1,240 people are in the west concourse. The operations lead looks through the glass at the west concourse, which is busy but nothing like 1,240 busy, and asks where the number came from. Nobody can answer beyond naming the supplier.

Computer vision people counting is the most capable of the anonymous counting methods and the easiest to misread, because the published accuracy figures are precise, real, and measured on crowds that look nothing like an exhibition hall. Understanding what those figures cover takes about twenty minutes and it changes what you ask for.

Two ways to count people in a frame

The first way is to detect people and count the detections. Run a detector, get a box per person, count the boxes. It works well when people are large in the frame and separated, and it collapses in a crowd, because a detector needs enough pixels on a body to make a decision and a dense scene does not supply them.

The second way skips individuals entirely. The model outputs a density map, one value per pixel describing the local concentration of people, and the count is the sum over the map. No box, no face, no identity, and a number that degrades gracefully as the crowd thickens.

Zhang, Zhou, Chen, Gao and Ma set the modern version of this out at the Computer Vision and Pattern Recognition conference in 2016, in a paper on multi-column convolutional networks running from page 589 to page 597. Their architecture used three parallel columns with different filter sizes, so that features learned in each column suit a different head size, which is the central difficulty in a photograph of a crowd: the person at the front might be 200 pixels tall and the person at the back 6. They also released the dataset the field then standardised on, 1,198 images carrying around 330,000 annotated heads, split into a congested Part A of 482 images collected from the internet and a sparser Part B of 716 images from streets in Shanghai.

What mean absolute error means, and what it means on your floor

Accuracy in this literature is a mean absolute error: average the size of the count error across the test images, in people.

Li, Zhang and Chen published CSRNet at the same conference in 2018, using dilated convolutions in place of the multi-column design, and their results table gives both models side by side. On Part A, MCNN reports a mean absolute error of 110.2 and CSRNet reports 68.2. On Part B, MCNN reports 26.4 and CSRNet 10.6.

Turn those into percentages, because a mean absolute error is meaningless without the size of the thing being counted. Part A's 482 images average around 500 people each. CSRNet's 68.2 against 500 is 13.6 per cent. MCNN's 110.2 against the same 500 is 22.0 per cent. The improvement between the two papers is 42.0 people, which is 38 per cent off the earlier error, and it took two years of the field's attention.

Now Part B. The two parts together hold about 330,000 heads and Part A holds roughly 241,000 of them, so subtracting leaves around 89,000 spread over 716 images, which is about 124 people per image. CSRNet's 10.6 against 124 is 8.5 per cent, and MCNN's 26.4 is 21 per cent.

That is the arithmetic that decides which figure you should quote. A concourse frame holding 900 people sits closer to Part A, and 68.2 on 900 is 7.6 per cent. An aisle frame holding 120 people sits in Part B territory, and applying a Part A error of 68 to a scene of 120 would be a 57 per cent error, which is why quoting the headline Part A number for a sparse hall camera is worse than useless.

One more column is worth reading. The same table gives a second error measure alongside the mean absolute error, in the same units, and CSRNet's is 115.0 on Part A against its mean absolute error of 68.2. A squared error summary running well above the absolute one says the error distribution has a tail: most frames are close and a few are badly wrong. On a live floor those few frames are not randomly distributed, and the next section is about why.

Why does the benchmark number flatter the supplier?

Because it was earned on the same data distribution it was measured on.

A model trained on ShanghaiTech and tested on ShanghaiTech is being asked about photographs taken with similar cameras, at similar angles, of similar crowds. Your hall is a different distribution in every one of those respects, and nothing in the published figure covers the move. A supplier quoting a benchmark mean absolute error is telling you how their model compares with other models on a shared test, which is a fair and useful thing to know and a different question from how it will do on your concourse.

The second reason is scene selection. Benchmark images are photographs somebody chose to annotate, mostly taken from high up with the crowd filling the frame. A hall camera is where the rigging plan allowed, usually looking along the hall at a shallow angle, with the far half of the frame at a resolution that puts four pixels on a head.

Ask for the error on frames from your own venue, annotated by hand, at the density and the camera angle you will actually run. Fifty frames is enough to see whether the model is in the right order of magnitude, and it costs a day of somebody counting heads in images.

What breaks on a show floor specifically

Four things, in rough order of how much damage they do.

Shallow camera angles. Density models cope with scale variation within a frame, and they cope less well when most of the frame is at a scale the training data barely contained. A truss-mounted camera looking down a 90 metre hall is mostly far field.

Stand builds and hanging banners. A retail street occludes people with other people. A hall occludes them with structures the model has never seen, and a double-decker stand removes a quarter of the floor from the frame entirely. The count for the region behind it is zero and it will be reported without a caveat.

Lighting that changes by the hour. Stand lighting, screens, spotlights on a feature area and daylight through a glazed concourse all move the exposure, and a hall at 08:00 is a different scene from the same hall at 15:00.

Frames against people. A density map gives you the count in one frame. It cannot give you how many different people passed through in an hour, because the same person appears in hundreds of frames. Turning frames into arrivals needs tracking or a counting line, which is a separate task with separate error, and a supplier who quotes counting accuracy for a throughput question has answered the wrong one. Directional throughput at a door is better served by a range sensor, and where lidar holds up and where crowds break it is C19.

Does counting without identifying change the privacy position?

It changes part of it, and the part it changes is worth understanding precisely.

The European Data Protection Board adopted version 2.0 of its guidelines on processing personal data through video devices on 29 January 2020. Their opening discussion separates video analytics techniques into ones that are "more intrusive (e.g. complex biometric technologies)" and ones that are "less intrusive (e.g. simple counting algorithms)", which puts density map counting at the gentler end of the range by the regulator's own framing.

The guidelines then set out three criteria for whether processing produces biometric data under Article 9: the nature of the data, the means and way of processing, and the purpose, where the purpose must be to uniquely identify a natural person. They go on to state that "when the purpose of the processing is for example to distinguish one category of people from another but not to uniquely identify anyone the processing does not fall under Article 9".

What that does not do is take you outside the GDPR. The frames going into the model are images of identifiable people and remain personal data whatever the model outputs. You still need a lawful basis, signage, a retention rule and, on most exhibition deployments, a data protection impact assessment. The practical consequence is a design decision: if the density map is the only thing retained and frames are discarded in memory, the record you hold is a set of counts, and that is a much easier conversation with a venue and with an exhibitor than a hard drive of footage. Design it that way deliberately, and write down that you did.

Where this stops

The honest limit is that an anonymous count has no join key. It can tell you 1,240 people were in the concourse and it cannot tell you whether they were attendees, exhibitor staff, contractors or the people from the conference next door. Every other measurement in an event data set hangs off a badge, and this one does not hang off anything.

The second limit is that camera coverage is a floorplan constraint dressed as a technical one. You will get cameras where rigging, power and network allow, which is rarely where the questions are. A count for the part of the hall you can see is not a count for the hall, and the gap between them is unmeasured unless you go and measure it.

Neither point argues against cameras. It argues for using them for the decisions their output actually supports, which is the sensor selection question in choosing an instrument per decision in C24, and for treating any anonymous device count with the same suspicion you already apply to probe request counts after address randomisation in C22. Both belong in the same attendee analytics plan and neither replaces a badge read.

Start by pulling twenty frames from your existing show cameras, spread across the day and the hall, and counting the heads in them by hand. Compare each with what the system reported for that timestamp. Twenty frames will not give you a confidence interval, and it will tell you within an afternoon whether the error you are living with is 8 per cent or 40.