An entry scan deduplication rule that survives a busy hall door

One entrance, three days, 34,100 rows in the reader log. The operations report said 34,100 entries. The registration system said 6,120 people had badges that were used at that door. Both figures were correct about what they measured and neither answered the question the show director had asked, which was how many people came in through that door on Tuesday.

Entry scan deduplication is the work that sits between those two numbers, and it takes two rules applied in order, doing two different jobs. Teams usually build one rule, argue about its length, and never notice that the argument is happening about the wrong parameter.

Why does one person produce four reads in four seconds?

Stand at a busy pedestal reader for ten minutes and you see every cause.

The reader double-fires. Proximity readers poll continuously, and a badge left hovering in the field for two seconds gets read twice, sometimes three times. This has nothing to do with the person.

The gate does not open. The attendee presents, nothing happens, they present again harder, then a steward waves them through. Three reads, one crossing.

The badge goes through twice for two people. One person taps, walks through, hands the badge back over the barrier. Two reads, two crossings, one credential.

Somebody steps outside. A phone call, a cigarette, a colleague arriving. Back in four minutes later. Two reads, two genuine crossings, one attendance by any sensible definition.

Those four causes need different treatment, which is why one window cannot do the job. The first two are reader artefacts and should disappear. The third is a policy question about badge sharing. The fourth is real movement that most reports want collapsed.

The debounce window, and how to size it

The first rule is a debounce: within a window of N seconds, the same badge at the same reader counts once. Per badge and per reader both matter. Per badge alone would merge a genuine crossing at hall one with a genuine crossing at hall four ninety seconds later, which is a real journey you may want.

Sixty seconds is the right default for a pedestal reader on a hall door. It swallows every double fire, it swallows the present-again-harder sequence, and it is short enough that stepping outside and coming back does not vanish.

Sizing it from your own data takes one query. Take the gaps between consecutive reads of the same badge at the same reader, and plot the distribution on a log scale. You will see two humps: a tight cluster under about four seconds, which is the reader talking to itself, and a broad spread from several minutes upwards, which is people. The valley between them is where the window belongs, and on most door hardware that valley sits somewhere between fifteen and ninety seconds.

Anything below thirty seconds leaves double fires in the file. Anything above two minutes starts eating real crossings at venues where the smoking area is ten steps outside the door.

The session window, which is the rule that actually decides your number

The second rule collapses the surviving reads into attendances: one attendance per person per day, per whatever boundary you have decided counts. This is the rule everybody should be arguing about, and it is usually the one nobody writes down.

Two things make it harder than it looks. It operates per person, so it needs an identity, and a badge is a proxy for a person that fails whenever badges are shared or reprinted. Somebody who loses a badge and gets a replacement at 14:00 has two credentials and will produce two attendances unless the reprint is linked back. Ask your registration system whether reprints carry the original registration identifier. Roughly half of them do.

The other difficulty is what counts as a day. A show that runs 09:00 to 18:00 has an easy answer. A show with an evening reception until 22:30 does not, and a calendar-day rule will count the reception separately from the afternoon on the floor. Use show-day boundaries defined by the opening hours, and store them as a table, because next year the hours will change.

A third detail decides more arguments than either window. Which reads are eligible for the rule at all depends on the source system and the device owner recorded against each row, and that labelling work is the scan taxonomy sitting underneath everything here. Feed session scans or exhibitor lead scans into an entry deduplication rule and it will produce a number nobody can name. Separating a genuine re-entry from a first arrival on the same day is a separate problem again, worked through in re-entry tracking.

Working the numbers on 34,100 reads

Here is the whole chain on that one entrance, across a three-day show.

Raw reads in the log: 34,100. Apply the sixty second debounce per badge per reader and 10,300 rows disappear, leaving 23,800. That is a 30.2 per cent reduction, which sounds alarming until you remember that a hesitating badge in a proximity field is three rows on its own.

Apply the session rule, one attendance per badge per show day, and 23,800 becomes 10,450 attendances. Count distinct badges across all three days and you get 6,120 people.

Three ratios fall out, and each one is worth keeping as a control chart across editions. Raw reads per attendance, 34,100 divided by 10,450, is 3.26. Attendances per person, 10,450 divided by 6,120, is 1.71 days each. Debounced reads per attendance, 23,800 divided by 10,450, is 2.28.

Now the part that surprises people. Move the debounce window to thirty seconds and the debounced count rises to 25,100. Move it to two minutes and it falls to 22,900. That is a swing of 2,200 rows, about 9.2 per cent of the middle figure. The attendance figure stays at 10,450 in all three cases, because the session rule absorbs everything the debounce leaves behind.

So the debounce length matters enormously for anything computed per read, which includes door throughput, reads per minute and the shape of your arrival curve. It is nearly irrelevant to attendance. Teams routinely spend a meeting on the debounce and no time at all on the session boundary, which is the reverse of the sensitivity.

How do you know the rule has not deleted real people?

Deduplication only removes rows, so every rule is a bet that the rows it removed were artefacts. The way to check the bet is a short audited window, and the arithmetic behind it is ninety years old.

Lincoln published the method in 1930 in United States Department of Agriculture Circular 118, estimating waterfowl abundance from banding returns. Two independent capture occasions, an overlap between them, and a population estimate: the number caught first times the number caught second, divided by the number caught both times. Chapman gave the bias-corrected version in 1951 in University of California Publications in Statistics, adding one to each term and subtracting one at the end, which behaves much better at small overlaps.

Run it on a door for ninety minutes. A steward writes down every badge number that passes. The reader logs its own. Then compare.

Steward recorded 412 individuals. Reader recorded 396. Both recorded 381. The Lincoln estimate is 412 times 396 divided by 381, which is 163,152 divided by 381, or 428.2. Chapman gives 413 times 397 divided by 382, minus one, which is 163,961 divided by 382 minus one, or 428.2 as well.

So about 428 people crossed that door in ninety minutes. The reader saw 396 of them, which is 92.5 per cent coverage. The steward saw 412, which is 96.3 per cent. Between them they missed one person.

Apply the reader coverage to the door's own day total and 9,400 recorded entries implies about 10,160 actual crossings. That gap is real, it is not caused by your deduplication rule, and it is the kind of thing that belongs in an attendee analytics report as a stated band rather than as a silent shortfall.

Where this stops

The capture-recapture check assumes the two observers fail independently. They do not. A group of six walking through together while a steward waves them past is missed by the reader and missed by the steward at the same moment, for the same reason. That correlation biases the estimate downwards, so treat 92.5 per cent as an optimistic reading of reader coverage.

The same estimator misbehaves badly if you try to apply it across two show days instead of two observers. Day one saw 4,980 distinct badges, day two saw 5,140, and 1,930 appeared on both, which gives an estimate of 13,260 people against 8,190 actually observed. That would imply 5,070 people attended without ever being scanned, which is nonsense. The number is telling you that attendees are not equally catchable across days, because most of them only ever intended to come once. The estimator is doing its job and the assumption is wrong.

The last limit is the shared badge. Deduplication by credential cannot see two people using one badge, and no window length changes that. Duplicate handling inside an exhibitor's own lead file has a different shape again, with different stakes, and it is worked through in duplicate badge scans.

Start this week by running the gap distribution query on your last edition's reader log, one door at a time. If the valley in that distribution sits somewhere other than where your current window is set, you have found a cheap correction to make before the next show.