Data quality.
Quality as an operating discipline. Freshness and volume checks, schema drift, duplicate rates, stewardship queues, data contracts, ownership and quality SLAs.
21 articles · Unified data
- The event data quality dimensions worth measuring on a registration fileevent data quality dimensionsThe six event data quality dimensions only work on a registration file once each one is bound to a named field. Six numbers from one show, with the arithmetic.
- Fitness for use data quality beats a generic accuracy targetfitness for use data qualityFitness for use data quality rates one registration file separately for each thing you do with it, so the exhibitor report and the pacing model get different verdicts.
- How to build a registration data quality score your team trustsregistration data quality scoreA registration data quality score is only trustworthy when the weights are published. A worked weighted average over four dimension pass rates, with the arithmetic.
- Data profiling a registration file before you trust a single numberdata profiling a registration fileData profiling a registration file means five statistics per column before any dashboard is built. The top twenty values on job title is the one that changes plans.
- Registration record completeness measured against what the post show report needsregistration record completenessRegistration record completeness measured against every form field gives a number nobody acts on. Measure against the fields your published outputs read instead.
- Which registration fields actually matter to the reports you publishwhich registration fields actually matterTo find which registration fields actually matter, count how many published outputs break without each one, then set targets only where that count reaches two.
- Completeness rate by field turns a vague data problem into a queuecompleteness rate by fieldThe completeness rate by field is the share of records holding a non null and non placeholder value. Placeholder detection is what separates a real rate from a flattering one.
- A weighted completeness score that reflects what each field is worthweighted completeness scoreA weighted completeness score rates each registration record by the value of the fields it holds, so a missing phone number stops costing as much as a missing company.
- Job title normalisation when every attendee types their own titlejob title normalisationJob title normalisation in three passes: strip case and punctuation, apply a synonym map, then join to a taxonomy. Most of the collapse happens before any semantic work.
- Deriving seniority level from job title text on a registration formseniority level from job titleDeriving seniority level from job title text needs an ordered keyword ladder run highest first, an exclusion pass before it, and the unknown share published alongside.
- Company name standardisation across registration and exhibitor systemscompany name standardisationCompany name standardisation is the step before any cross system comparison: case, whitespace, legal suffixes from a published list, and the raw string kept.
- Country field normalisation and the two letter codes that end the argumentcountry field normalisationCountry field normalisation maps UK, U.K., England and GB to one ISO 3166-1 alpha-2 code on ingest, keeps the original, and counts unmapped values as a metric.
- State and province codes are the quiet failure in geographic reportingstate and province codesState and province codes collide across countries unless they carry a country prefix. Store ISO 3166-2 subdivision codes and validate the prefix against the country.
- Postal code validation catches errors that country level checks never seepostal code validationPostal code validation tests each postcode against a pattern chosen by country code, reports a pass rate per country, and turns a vague data problem into one form fix.
- Data freshness monitoring for a business whose data arrives in burstsdata freshness monitoringData freshness monitoring on an events business needs a threshold that moves with the show calendar, because one fixed window pages you all summer for nothing.
- Freshness checks for annual feeds when a source lands once a yearfreshness checks for annual feedsFreshness checks for annual feeds fail when built on hours. Replace the clock with the event calendar so the check asserts a file landed within days of show close.
- Stale event data detection and the age at which a record misleadsstale event data detectionStale event data detection sets a maximum age per downstream use, not per table, then reports the share of records past it so a call list stops being built on 2019.
- Data volume anomaly detection on a business with one peak a yeardata volume anomaly detectionData volume anomaly detection on an events business builds its expected range from prior editions at the same number of days before show open, then alerts outside it.
- Row count checks on seasonal data built from the prior year curverow count checks on seasonal dataRow count checks on seasonal data work once you decompose the series first: run STL on daily counts, keep the remainder, and put the control limit on that.
- Zero row load detection when an empty file is sometimes correctzero row load detectionZero row load detection needs a gate on the show calendar, because zero registrations 200 days out is fine and zero on the Tuesday of show week is an incident.
- Schema drift detection for the exports you did not write and cannot controlschema drift detectionSchema drift detection on a registration export you do not control: hash the ordered column list and types on every load, and decide in advance what stops it.