Fitness for use data quality beats a generic accuracy target

Two people looked at the same registration export in the same week and reached opposite conclusions about it. The analyst running the pacing model said the file was fine. The person building the geographic breakdown for the exhibitor report said it was unusable.

Both were right, and the fitness for use data quality framing is what makes that a sensible sentence instead of a contradiction. Quality here is a property of the pairing between a data set and a task, so a single file gets as many verdicts as it has uses.

That sounds like a semantic dodge until you try to run a data quality programme without it. Then you find yourself setting a target like ninety-five per cent accuracy across the registration file, and discovering that nobody can act on it, because the fields that would need fixing to reach it are not the fields anybody is blocked on.

What fitness for use actually claims

The position comes out of the total data quality management work at MIT in the mid-nineties. Wang and Strong published the framework in the Journal of Management Information Systems in 1996 under the title Beyond Accuracy, and Strong, Lee and Wang restated it in Communications of the ACM in May 1997 after studying 42 data quality projects at three organisations: an international airline, a hospital and a health maintenance organisation.

Their definition is one sentence. High quality data is data that is fit for use by data consumers. The consequence they draw is the interesting part: quality cannot be assessed independent of the people who use the data, so a purely intrinsic view of quality, one that treats a defect as an error in stored data and nothing else, will miss the problems that actually stop work.

They sort the characteristics of high quality data into four categories, and the table is worth having in front of you when you argue about a registration file.

Intrinsic quality covers accuracy, objectivity, believability and reputation. These are the ones people mean when they say the data is bad.

Contextual quality covers relevancy, value-added, timeliness, completeness and amount of data. Every one of these is defined against a task, which is the whole point.

Representational quality covers interpretability, ease of understanding, concise representation and consistent representation.

Accessibility quality covers accessibility and access security.

The standards world arrives at a compatible answer from a different direction. ISO 9000:2015 defines quality as the degree to which inherent characteristics of an object meet requirements, quoted in that form by Black and Van Nederpelt in the DAMA NL data quality dimensions report of 2020. Requirements come from somewhere, and on a registration file they come from whoever is going to use it.

Why does the same file pass for one use and fail for another?

Because the two uses read different columns and tolerate different failures, and a file-level score averages over exactly that difference.

Take one edition, 41,800 registration rows.

The pacing model reads three things: the created timestamp, the registration type, and the acquisition channel. 210 rows have a null created timestamp because they came in through a partner import that lost it. That leaves 41,590 usable rows, or 99.5 per cent of the file. The model runs. Its weekly curve is not visibly disturbed by 210 rows out of 41,800, and no decision it feeds would change if they were recovered.

The geographic breakdown in the exhibitor report reads country. 6,400 rows have no country value at all, though most of them carry a postcode. Every one of those rows has to be dropped or bucketed as unknown, so the usable base is 35,400 rows, or 84.7 per cent of the file. An unknown bucket holding more than fifteen per cent of the audience is not something you put in front of exhibitors who are deciding whether the show reaches their market.

One file. 99.5 per cent fit for one use, 84.7 per cent fit for another. A single number describing the quality of that file would have to sit somewhere between the two, and wherever it sat it would be wrong for both.

The postcode with no country

The 6,400 rows are worth looking at closely, because they are the clearest example of the contextual category doing work that an accuracy check cannot.

Of those 6,400 rows, 4,100 hold a purely numeric five-digit postcode. That is 64.1 per cent of them. Every one of those values is exactly what the registrant typed. Not one of them is inaccurate in any sense a proofreader would recognise. 75008 is a real postcode in France. It is also a valid format in the United States, Germany, Spain, Italy and Finland, and the row does not say which.

Intrinsic quality on that field is perfect. Contextual quality is zero, because the task needs a country and the field cannot supply one without a companion value that is missing.

This is why I find the four categories more useful in practice than a flat list of dimensions. A flat list makes you ask whether the postcode is right. The categories make you ask whether the postcode is right for what somebody is about to do with it, and on an event file the second question has a different answer often enough to matter.

The fix is a form change and a backfill, and the fix is cheap. What is expensive is the year you spend not noticing, because the field looked full and the null rate on postcode was excellent.

Rating a file against each use

The method is short enough to run on a Friday.

List the outputs that leave the building. For most organisers that is four things: the post-show report, the exhibitor scorecard, the pacing model, and whatever the sales team opens when it calls an account.

For each one, write down the columns it reads. Not the columns it could read. The columns in the query.

For each output, compute the share of rows that carry a usable value in every column that output reads. That single number is the fitness of the file for that use, and it is defensible because you can name the query it came from. The mechanics of counting a record as complete against a named output, and the trap of counting against the whole form instead, are worked through in completeness measured against the report.

Then write the four numbers down next to each other and look at the spread. On the file above it runs from 99.5 to 84.7, and the spread is the finding. A narrow spread means one shared defect and one fix. A wide spread means the problem belongs to one output and one owner, which is much better news than a file-wide quality problem.

If you want to go further, rate each use on the four categories rather than one blended figure. The exhibitor scorecard usually scores well on intrinsic and badly on contextual. The sales workspace usually scores well on both and badly on accessibility, because the data is correct and sitting in a system the sales team cannot open.

What is wrong with a single accuracy target?

It sets an objective nobody can decompose into work.

Say the target is 95 per cent accuracy on the registration file. Somebody has to decide which fields count, and every choice changes the number. Include phone, which is optional and 41 per cent populated, and the target is unreachable. Exclude it, and you have quietly declared phone unimportant without anyone agreeing to that. The target is now a negotiation about scope disguised as a measurement.

A per-use target avoids that. The geographic breakdown needs country on 95 per cent of rows. That is a sentence with an owner, a query, a number today, and a definition of done. Whether phone is populated does not enter into it.

There is a second problem with the file-wide target, which is that it prices every field the same. A file-wide score treats a missing phone number and a missing badge identifier as one defect each, and the badge identifier failure breaks four reports while the phone failure breaks none. Once you accept that the fields are worth different amounts, you are building a weighted measure, and the weights have to be published rather than buried.

None of this argues against measuring the dimensions on the file itself. Binding each of the six dimensions to a named field gives you the raw material. Fitness for use decides which of those numbers anyone should care about this quarter.

Where this stops

Rating per use has an obvious failure mode, which is that you rate the uses you know about. The registration file also feeds things nobody listed: an ad hoc extract somebody sent a media partner, a list pulled for a webinar, the file a new analyst joins to last year's export. Those uses have their own fitness, nobody measured it, and the first sign of trouble is a number in a deck that disagrees with yours.

The framework also gives you no way to compare across uses. Is 84.7 per cent fitness for the geographic breakdown worse than 91 per cent fitness for the sales workspace? The question has no answer inside the method, because the two numbers have different denominators and different consequences. You need a business judgement about which output matters more, and the framework deliberately does not supply one.

And there is a cost to running it this way. A per-use rating has to be recomputed when a query changes, which means it is only as current as your knowledge of what the queries read. A team that rewrites its post-show report every year will find last year's fitness ratings quietly obsolete. Tying the rating to a governed definition of the record rather than to a particular query helps, though it does not remove the maintenance.

Take the query behind your geographic breakdown, list the columns in its where clause and its group by, and count the rows in this year's file that carry a usable value in all of them. That single fraction is your file's fitness for the one output exhibitors read most closely, and you will have it before lunch.