What belongs in a data contract for an event data feed
Six clauses cover most event data feeds: the primary key and its uniqueness guarantee, the required columns with types, the arrival window, the allowed null rate per column, the notice period for schema changes, and a named owner on each side. Write them as YAML beside the loader so a test can read them.
The word contract makes people reach for language, and language is what kills these documents. A page that says the vendor will provide accurate and timely registration data has no clause anybody can test, so nothing gets checked, and the first person to inherit the file will assume it was decorative.
What belongs in a data contract is whatever a piece of software can check on the morning a file arrives. For an event feed, six clauses cover most of it, and each one maps to an assertion you can run before the load.
Six clauses cover most event feeds
- The primary key and its guarantee. Which column identifies a row, and the promise that it is unique within an event edition and stable between exports.
- The required columns and their types. The subset your reports reference, each with the type it arrives as and the format where the type alone is ambiguous.
- The arrival window. The time by which the file must land, expressed against a schedule that moves with the show calendar.
- The allowed null rate. Per column, the share of rows that may be empty before the clause fails.
- The notice period for schema changes. How many days before a change, and by what route the notice arrives.
- The named owner on both sides. One person at the vendor, one on your team, both by name.
For comparison, the Open Data Contract Standard, the YAML specification maintained by the Bitol project under the LF AI and Data Foundation, defines eleven sections at version 3.1.0: fundamentals, schema, references, data quality, support and communication channels, pricing, team, roles, service level agreement, infrastructure and servers, and custom properties. That is the right surface area for a standard that has to describe any data product anywhere. The six above are what a two person event data team can write this week and actually maintain, and they map onto the standard's schema, data quality, SLA and team sections when you outgrow the short version.
What does the primary key clause have to promise?
Two things, and teams routinely ask for only the first.
Uniqueness is the obvious one. Within one event edition, one registration id appears once. That is a single assertion and it fails loudly.
Stability is the one that gets forgotten and costs more. The registration id for a given registration must be the same in tomorrow's export as it was in yesterday's. Plenty of platforms generate export row identifiers at export time, which look like keys, behave like keys for exactly one file, and make incremental loading impossible. If the vendor cannot promise stability, you need to know now, because every downstream design decision changes: you are then reloading the full file each time and reconciling by content, which is slower and needs its own duplicate handling.
Write both into the clause explicitly. Unique within edition, stable across exports, and never reused after a registration is cancelled, which is the third case and the one that produces a cancelled record silently inheriting a new person's details.
The arrival window and what on sale means
An arrival window written as a fixed daily time will be broken half the year, because event data does not arrive at a constant rate.
Write it against the show calendar. The file lands by 08:00 local on each day between the on-sale date and 14 days after show close, and weekly on the Monday outside that period. Both halves need a definition somebody can look up: on-sale date means the date public registration opens, which is a field in your own event table, and show close is the last day of the event.
Say what a miss means as well. A file arriving at 11:00 rather than 08:00 is a late file, and if the pacing report runs at 09:00 the whole day's numbers are yesterday's. Across a 120 day on-sale period, a vendor missing the window on 3 per cent of days misses it about 4 times, and if two of those fall in the final fortnight before the show they matter far more than the other two. Counting misses by period rather than as an annual average is the version worth reporting.
How strict should the null clause be?
Set it from the observed rate, then work out what the threshold means in rows.
Suppose the daily registration file carries 12,400 rows and you write the country column at 98 per cent complete. The clause allows 12,400 times 0.02, which is 248 nulls. A file arriving with 310 nulls fails by 62 rows, and the question for the person who gets that alert is whether 62 rows matter, which they can now answer because the failure is expressed as a count rather than as a percentage that reads like a rounding error.
Great Expectations expresses this directly. The expectation expect_column_values_to_not_be_null takes a mostly parameter, documented as a float between 0 and 1 where the check succeeds if at least that fraction of values match, so the clause above is mostly 0.98. The dbt equivalent is the built in not_null test with a threshold config on top.
Where teams go wrong is writing 100 per cent for everything. A country clause at 100 per cent on a form where country is optional will fail every single day, and a check that always fails is a check somebody switches off in week two. Take 90 days of history, look at the actual completeness distribution per column, and set the clause a little below the worst normal day.
Writing it as YAML next to the loader
Put the six clauses in a file the loader reads, in the same repository, reviewed like code.
source: acme_registration_export
owner_vendor: [email protected]
owner_internal: [email protected]
key:
column: registration_id
unique_within: event_edition_id
stable_across_exports: true
required_columns:
registration_id: string
registered_at: timestamp
email: string
company: string
country: string
badge_type: string
arrival:
on_sale_period: daily by 08:00 local
quiet_period: weekly Monday by 08:00 local
null_rate_max:
country: 0.02
company: 0.05
badge_type: 0.00
schema_change_notice_days: 30
That file is about thirty lines and it does two jobs. The test suite reads it, so nobody maintains the clauses in one place and the assertions in another. And a person reads it, which matters more, because the next data hire can learn what this feed is supposed to be in under a minute.
The two names at the bottom
Every clause needs somebody who is accountable when it breaks, and that has to be two named people rather than two departments.
Vendor side, ask for the name of the person who will receive the schema change question, and expect it to be a support queue. Push for a named technical account contact if the feed carries a show you cannot run without. Internal side, name the person who clears a held file, and name a second one, because the first will be on a stand somewhere the day it matters.
Put both names in the YAML. When somebody leaves, the commit that changes the name is the record that the handover happened.
Where a clause stops being testable
The null rate clause breaks on small files. A quiet month export of 40 rows with a 98 per cent country clause allows 40 times 0.02, which is 0.8 nulls, so a single missing country fails the file. That is a real failure that means nothing, and it will train the team to ignore the channel within a month.
The fix is a floor: the clause fails when the null count exceeds the greater of the percentage allowance and five rows, and it does not run at all below a few hundred rows. Write the floor into the contract, because a threshold that only holds at scale is a threshold you have to explain every off season.
Two other clauses look testable and are not. Stability of the key cannot be checked from one file, only across two, so it needs the previous export retained and compared, which is a design decision at the storage layer. And the notice period is not testable at all by software: nothing in the file says whether you were told thirty days ago. Somebody has to record the notice when it arrives, which means the contract needs a log with dates in it, and that log is the only evidence you will have in a renewal conversation.
Write the six clauses for one feed today, in one YAML file, and leave the assertions for later. The clauses are the part that requires agreement and judgement, and once they exist, turning them into checks that run at the boundary is mostly mechanical, which is K29's subject. Then take the same file to the vendor, since the negotiation is a separate skill from the drafting and K27 covers it. Grading what happens when one of these clauses is breached mid season belongs with the severity ladder in K23, and the whole set is what makes the rest of your unified data work reproducible.
Questions people ask about what belongs in a data contract
- What belongs in a data contract for a registration feed?
- Six clauses do most of the work. The primary key and the promise that it is unique and stable, the required columns with their types, the window the file must arrive in, the share of nulls tolerated per column, the notice period before any of it changes, and the person accountable on each side.
- Should a data contract list every column in the feed?
- No. List the columns your reports reference and would stop work over. A contract covering every column the vendor happens to send is a description of one product rather than a statement of your requirements, and it becomes useless the moment the platform is replaced.
- How do you write a data contract so it can be tested?
- Put it in a machine readable file, normally YAML, in the same repository as the loader. Each clause maps to an assertion: uniqueness on the key, presence and type per required column, a timestamp check for the arrival window, and a null fraction threshold per column, which most testing tools express directly.
Related reading
- Data contracts with registration vendors that survive a change of platform
- Enforcing a data contract without a lawyer and without breaking the show
- How to spot a breaking change in a data feed early