Data quality.
Quality as an operating discipline. Freshness and volume checks, schema drift, duplicate rates, stewardship queues, data contracts, ownership and quality SLAs.
31 articles · Unified data
- The event data quality dimensions worth measuring on a registration fileevent data quality dimensionsThe six event data quality dimensions only work on a registration file once each one is bound to a named field. Six numbers from one show, with the arithmetic.
- Fitness for use data quality beats a generic accuracy targetfitness for use data qualityFitness for use data quality rates one registration file separately for each thing you do with it, so the exhibitor report and the pacing model get different verdicts.
- How to build a registration data quality score your team trustsregistration data quality scoreA registration data quality score is only trustworthy when the weights are published. A worked weighted average over four dimension pass rates, with the arithmetic.
- Data profiling a registration file before you trust a single numberdata profiling a registration fileData profiling a registration file means five statistics per column before any dashboard is built. The top twenty values on job title is the one that changes plans.
- Registration record completeness measured against what the post show report needsregistration record completenessRegistration record completeness measured against every form field gives a number nobody acts on. Measure against the fields your published outputs read instead.
- Which registration fields actually matter to the reports you publishwhich registration fields actually matterTo find which registration fields actually matter, count how many published outputs break without each one, then set targets only where that count reaches two.
- Completeness rate by field turns a vague data problem into a queuecompleteness rate by fieldThe completeness rate by field is the share of records holding a non null and non placeholder value. Placeholder detection is what separates a real rate from a flattering one.
- A weighted completeness score that reflects what each field is worthweighted completeness scoreA weighted completeness score rates each registration record by the value of the fields it holds, so a missing phone number stops costing as much as a missing company.
- Job title normalisation when every attendee types their own titlejob title normalisationJob title normalisation in three passes: strip case and punctuation, apply a synonym map, then join to a taxonomy. Most of the collapse happens before any semantic work.
- Deriving seniority level from job title text on a registration formseniority level from job titleDeriving seniority level from job title text needs an ordered keyword ladder run highest first, an exclusion pass before it, and the unknown share published alongside.
- Company name standardisation across registration and exhibitor systemscompany name standardisationCompany name standardisation is the step before any cross system comparison: case, whitespace, legal suffixes from a published list, and the raw string kept.
- Country field normalisation and the two letter codes that end the argumentcountry field normalisationCountry field normalisation maps UK, U.K., England and GB to one ISO 3166-1 alpha-2 code on ingest, keeps the original, and counts unmapped values as a metric.
- State and province codes are the quiet failure in geographic reportingstate and province codesState and province codes collide across countries unless they carry a country prefix. Store ISO 3166-2 subdivision codes and validate the prefix against the country.
- Postal code validation catches errors that country level checks never seepostal code validationPostal code validation tests each postcode against a pattern chosen by country code, reports a pass rate per country, and turns a vague data problem into one form fix.
- Data freshness monitoring for a business whose data arrives in burstsdata freshness monitoringData freshness monitoring on an events business needs a threshold that moves with the show calendar, because one fixed window pages you all summer for nothing.
- Freshness checks for annual feeds when a source lands once a yearfreshness checks for annual feedsFreshness checks for annual feeds fail when built on hours. Replace the clock with the event calendar so the check asserts a file landed within days of show close.
- Stale event data detection and the age at which a record misleadsstale event data detectionStale event data detection sets a maximum age per downstream use, not per table, then reports the share of records past it so a call list stops being built on 2019.
- Data volume anomaly detection on a business with one peak a yeardata volume anomaly detectionData volume anomaly detection on an events business builds its expected range from prior editions at the same number of days before show open, then alerts outside it.
- Row count checks on seasonal data built from the prior year curverow count checks on seasonal dataRow count checks on seasonal data work once you decompose the series first: run STL on daily counts, keep the remainder, and put the control limit on that.
- Zero row load detection when an empty file is sometimes correctzero row load detectionZero row load detection needs a gate on the show calendar, because zero registrations 200 days out is fine and zero on the Tuesday of show week is an incident.
- Schema drift detection for the exports you did not write and cannot controlschema drift detectionSchema drift detection on a registration export you do not control: hash the ordered column list and types on every load, and decide in advance what stops it.
- Vendor export column changes and how to find them before a report breaksvendor export column changesVendor export column changes break reports quietly. Keep a per source column manifest, diff it on arrival, and route the difference to a person before the transform runs.
- How to spot a breaking change in a data feed earlybreaking change in a data feedA breaking change in a data feed is one that changes the result of a query you already have. Grade every change by blast radius before deciding who to page.
- Duplicate rate as a metric you can actually report to a boardduplicate rate as a metricDuplicate rate as a metric survives a board meeting only if you fix the definition of a duplicate first. Three defensible definitions give three numbers.
- False merge rate is the number your stewards should watch most closelyfalse merge rateFalse merge rate measures how often two different people were collapsed into one record. Sample 200 auto merged pairs, label each, and read the interval honestly.
- Which source creates duplicates and how to prove it with a simple querywhich source creates duplicatesTo find which source creates duplicates, tag every record at ingest and group the duplicate clusters by the set of systems they span. The signature names the culprit.
- Data contracts with registration vendors that survive a change of platformdata contracts with registration vendorsData contracts with registration vendors outlast the platform when they are written against your requirements: the key, the required columns, the arrival window and notice.
- What belongs in a data contract for an event data feedwhat belongs in a data contractWhat belongs in a data contract for an event feed: the key, the required columns and types, the arrival window, the allowed null rate, the notice period and two names.
- Enforcing a data contract without a lawyer and without breaking the showenforcing a data contractEnforcing a data contract means running the clauses as assertions at the boundary, before the load, and deciding in advance whether a failure quarantines the file.
- Designing a data stewardship queue that a human can finish each morningdata stewardship queueA data stewardship queue only works if it empties. Rank candidates by expected value, cap the daily list at what one person clears, and let the remainder expire visibly.
- Steward review capacity and the arithmetic of how many records fitsteward review capacitySteward review capacity is arithmetic: measured seconds per decision times hours per day. Work out how many candidate records one person clears before doors open.