How to build a registration data quality score your team trusts

Two shows in the same portfolio both came back at 91 on the data quality scorecard, and the operations lead treated them as equivalent. They had nothing in common. One had a stale export and clean fields. The other had current data with a country column that was a quarter free text.

That is the standing risk with a registration data quality score. A single number is easy to track and easy to put in a board pack, and it does its job only if the reader can pull it apart. So build it so it comes apart.

The three functional forms, and the one you need

Pipino, Lee and Wang wrote the short paper that most data quality metrics quietly descend from, in Communications of the ACM in April 2002. They name three pervasive functional forms for objective quality metrics: the simple ratio, the minimum or maximum operation, and the weighted average.

The simple ratio measures desired outcomes over total outcomes, and they note a preference worth copying. Count the exceptions, divide by the total, and subtract from one, so the metric follows the convention that 1 is the most desirable score and 0 the least. Most dimension-level measurements on a registration file take this form: rows failing a country validity test over total rows, subtracted from one.

The minimum operation assigns to a composite no higher a value than its weakest component. They call it conservative, which it is, and it is the form nobody in events uses and most of them should.

The weighted average is the one everybody reaches for. For it to behave, the paper is explicit: each weighting factor should be between zero and one, and the weighting factors should add to one. That single constraint is what keeps the composite on the same scale as the things it combines, and it is the first thing to check when somebody hands you a score you did not build.

Weights that add to one, and why it matters

If your four weights sum to 1.15, a file that is perfect on every dimension scores 1.15, and every comparison you make afterwards is against a moving ceiling. It sounds like a technicality. It is the reason a lot of vendor quality scores cannot be reconciled with anything.

Here is the set I would start from on a registration file, with the reasoning attached, and I want to be clear that these are a starting position rather than a finding.

Completeness gets 0.35, because a missing value stops a report rather than degrading it, and because completeness failures cluster in exactly the fields that carry the audience breakdowns exhibitors read.

Validity gets 0.25. An invalid country code is recoverable by normalisation, which makes it cheaper to fix than a missing value, but it fails silently in a group by, which makes it more dangerous than it looks.

Uniqueness gets 0.25. Duplicates inflate every count in the post-show report and they compound, because last year's duplicates are still there.

Timeliness gets 0.15, and this is the weight I have the least confidence in. It is low because a stale file is usually recoverable by re-running an export. It should probably be higher on any show where the export runs before the doors close.

Accuracy is absent, because on most registration files it is unmeasured, and giving a weight to a dimension you do not measure means either assuming a value or quietly renormalising the others. Both are worse than leaving it out and saying so.

Whatever set you choose, the weights go in the same document as the score, with a date on them. A score whose weights live in somebody's notebook is a number nobody measured. Keep the weight table wherever the record definitions live, so one version serves every show in the portfolio instead of each show drifting to its own.

The arithmetic on one show

An edition with 26,450 registration rows. Four dimension pass rates, each computed as a simple ratio and each with its own definition written down. The per field mechanics behind the completeness figure are in the pass rate you compute for every column.

Completeness comes in at 0.881. Validity, tested on country against ISO 3166-1 alpha-2, at 0.962. Uniqueness on badge identifier at 0.994. Timeliness, the share of registrations existing at show close that appear in the file, at 0.734, because the export ran early.

The weighted average is 0.35 times 0.881, plus 0.25 times 0.962, plus 0.25 times 0.994, plus 0.15 times 0.734.

That is 0.308 plus 0.241 plus 0.249 plus 0.110, which totals 0.907. The show scores 90.7.

Now the second show in the portfolio, 19,200 rows. Completeness 0.964, validity 0.812, uniqueness 0.999, timeliness 0.968.

0.35 times 0.964 is 0.337. 0.25 times 0.812 is 0.203. 0.25 times 0.999 is 0.250. 0.15 times 0.968 is 0.145. The total is 0.935, so that show scores 93.5.

Under three points between them, and the two files need completely different work. The first has a scheduling problem that one line in a cron file will fix. The second has a country column in four formats and a form that lets people type into it.

What happens when you change the weights?

Test it, because the answer tells you how much of the ranking is coming from the data and how much from your opinion.

Move completeness down from 0.35 to 0.20 and give the freed 0.15 to timeliness, taking it to 0.30. Nothing about either file changes.

The first show becomes 0.20 times 0.881, plus 0.25 times 0.962, plus 0.25 times 0.994, plus 0.30 times 0.734. That is 0.176 plus 0.241 plus 0.249 plus 0.220, or 0.885. It has dropped 2.2 points.

The second show becomes 0.193 plus 0.203 plus 0.250 plus 0.290, or 0.936. It has moved by a tenth of a point.

So a defensible change to one weight moves one show by 2.2 points and leaves the other where it was. If your quarterly target is a two point improvement, a weight revision could deliver it without anybody touching the data. That is the argument for versioning the weights and for republishing the prior period under the new weights whenever they change.

Report the minimum next to the average

The weighted average on the first show is 90.7. Its lowest component is 0.734.

The 90.7 is what goes in the board pack. The 0.734 is what determines whether the geographic breakdown in the post-show report can be published, because a file missing 26.6 per cent of the registrations that existed at close will understate onsite attendance no matter how clean its other columns are.

This is the min operation from the 2002 paper, used alongside the weighted average rather than instead of it. The average tells you how the file is trending. The minimum tells you what is currently blocked. Two columns on the same row of the scorecard, and the second one is the one with an owner's name against it.

I would go further and say the minimum is the number to put a threshold on. A rule that fires when any single dimension drops below 0.80 catches the first show immediately. A rule that fires when the composite drops below 0.85 never fires at all, because 0.907 has plenty of room above it, and the weighted average absorbs one bad dimension by design.

What should a data quality score actually be used for?

Tracking one file over time. That is the use it is good at, because the definitions and the weights are held constant and the only thing moving is the data.

Comparing two shows is where it starts to mislead, for the reason the two examples show. Comparing across organisers is worse, because the definitions will differ in ways nobody documents. If somebody quotes you a quality score for a registration file they did not build, the useful question is which four things it averages and what the weights are, and the answer is usually not available.

The score is also poor at deciding what to fix. It ranks nothing, because every dimension is aggregated across all fields before the weighting happens. A 0.881 completeness rate could be one field at zero or eleven fields at ninety per cent, and the work implied by those two is not comparable. That decomposition is a different measurement, and it is what scoring a record against weighted fields is for.

Where this stops

The score inherits every definition underneath it, and the definitions are where the disagreements actually live. If two people compute uniqueness on different keys they will get different pass rates, and the composite will hide the disagreement inside a number that looks arithmetically solid. Binding each dimension to a named field before any of this is the step that makes the composite mean anything, and skipping it produces a score with four arguments buried inside.

The weights are a judgement and they will stay a judgement. ISO 9000:2015 defines quality as the degree to which inherent characteristics of an object meet requirements, quoted in that form by Black and Van Nederpelt in the DAMA NL dimensions report of 2020, and requirements come from people. There is no procedure that derives 0.35 for completeness from the data. What you can do is state the weight, date it, and revise it when a low-weighted dimension breaks something, which is a slower and more honest way to converge than arguing about it in advance.

The last limit is that a composite invites exactly the behaviour it should discourage. Once a number is on a scorecard, somebody will try to move it, and the cheapest way to move a weighted average is to improve the highest-weighted dimension in the place where it is easiest, which is rarely where the damage is. Publishing the components next to the total is the only defence I have found that works.

Take your last two closed shows, compute the four pass rates on each, and write the eight numbers in a single table with the weights above them. You will know within an hour whether your portfolio has one data quality problem or two, and that is a different conversation from the one a single score starts.