Skip to content

Golden record construction after the merge decision has already been made

Unified dataUpdated 2026-08-238 min read

In short

Golden record construction assembles a single record for one resolved identity from the values that survived per attribute, and stores provenance beside each field. Every attribute carries the source row it came from, the system that supplied it and the timestamp it was observed, so the record can be explained and rebuilt from its inputs.

The finance lead reads an audience figure that depends on a job title field, asks where one value came from, and the answer takes four days. The value is on the golden record. The golden record was built by a nightly job. The job reads four source tables and applies rules that live in code, and reconstructing which of the four rows supplied that particular string means rerunning the job with logging turned up.

Golden record construction is the step where you either make that question cheap or make it expensive, and the decision costs about the same either way at build time.

What has already been settled, and what has not

By the time this step runs, two things are known. A set of source rows has been judged to describe one entity, and for each attribute a rule has chosen a winning value. Which rule and why is the subject of survivorship rules in J26.

What remains is assembly, and the literature has a precise name for it. Dong and Naumann's 2009 tutorial at the VLDB conference describes data fusion as "fusing records on the same real-world entity into a single record and resolving possible conflicts from different data sources", and separates two kinds of conflict worth keeping distinct in an event file. "Uncertainty is a conflict between a non-null value and one or more null values that are all used to describe the same property of a real-world entity." "Contradiction is a conflict between two or more different non-null values that are all used to describe the same property of the same entity."

Registration data produces far more of the first than the second. Most fields on most rows are empty, and the majority of assembly work is choosing between a value and a series of blanks. That distinction changes what the record should record. Filling a blank from another row is an act of completion, and overwriting one non empty value with another is an act of arbitration, and a reader deserves to know which one produced any given field.

Four registrations, eight fields

Take one buyer with four rows across the portfolio: a 2022 web registration at one show, a 2023 exhibitor guest list upload at another, a 2025 lead capture scan, and a 2026 web registration at a third show.

FieldSurviving valueSourceObserved
First nameMiguel2022 web registration2022-05-09
SurnameSantos Ferreira2026 web registration2026-01-18
Email[email protected]2026 web registration2026-01-18
Mobile+3519120004562025 lead capture scan2025-02-03
Job titleOperations Director2026 web registration2026-01-18
CompanyBrightline Logistica2026 web registration2026-01-18
Company domainbrightline.ptderived from email2026-01-18
CountryPT2023 guest list upload2023-11-21

Read the source column rather than the value column. The newest registration supplies four of the eight fields. The oldest supplies the first name, because the 2025 scan recorded Mike and the nickname rule rejected it. The only phone number in the entire history came off a lead capture scan in 2025, and the country came from a guest list upload three years old that nobody would think to look at.

One field is derived rather than sourced. The company domain was computed from the surviving email, so its provenance points at a rule and at the field it was computed from, and if the email is later corrected the domain has to be recomputed. Derived fields on a golden record need marking as derived, or somebody will eventually treat one as an observation.

Why does per field provenance matter more than record level provenance?

Database research made this distinction thirty years before most master data tools did. Buneman, Khanna and Tan, at the International Conference on Database Theory in 2001, separated why-provenance, which "refers to the source data that had some influence on the existence of the data", from where-provenance, which "refers to the location(s) in the source databases from which the data was extracted".

A golden record needs both, and they answer different questions. Why-provenance is the merge decision: these four rows are here because a matcher scored them above a threshold. Where-provenance is the field level fact: this job title string was copied from that row of that table.

Record level provenance gives you only the first. It tells you which rows went into the record, which means answering a question about one field requires re-deriving the whole thing. Storing where-provenance per field turns a four day investigation into a lookup, and it is the difference between a data team that can answer questions about its own output and one that can only regenerate it.

Dong and Naumann name this as unfinished business in their own field, listing the better inclusion of data lineage among the open problems for data fusion systems. Sixteen years on, most commercial pipelines still discard it at assembly time.

What do the provenance columns actually cost?

The objection is always storage, so do the arithmetic once.

Four provenance items per field, at roughly 20 bytes together: an 8 byte source row identifier, a 2 byte system code, an 8 byte timestamp and a 2 byte rule identifier. Eight fields on the record makes 160 bytes of provenance per golden record.

A large portfolio with 1.2 million resolved people therefore carries 192 million bytes, about 192 megabytes, to make every field on every record explainable. That is smaller than a single edition's badge scan table on most shows, and it costs nothing to query because nothing reads it until somebody asks a question.

The real cost is different and worth naming. Provenance has to be written by the same code that chooses the value, in the same transaction, or it drifts. A separate logging step that runs afterwards will eventually record what the rule would have chosen rather than what it did, and a provenance record that is sometimes wrong is worse than none, because it will be believed.

The golden record is a derived object

The temptation, once the record exists, is to let people edit it. Somebody in audience operations sees a wrong job title on the golden record, fixes it there, and the next load overwrites the fix.

Hold the line that the golden record is derived from the source rows and the cluster assignment, and that the only way to change it is to change one of those two inputs. A correction becomes a new row in a corrections source with its own timestamp and author, which then participates in survivorship like any other source and usually wins on recency. The correction survives every rebuild, it is attributable, and it can be reversed.

That design has a second benefit, which is that the whole record set can be rebuilt when a rule changes. Improve the nickname table and every affected golden record can be regenerated and compared against the previous version, so you can count how many records moved before deciding to ship. Without derivability, a rule change is an irreversible migration and teams stop making them, which is how a survivorship configuration written in a hurry in year one is still running in year five. Keeping the inputs intact is merge and unmerge operations in J30, and the same immutability is what makes rebuilds safe.

Which fields belong on the record at all

Not every column in a registration file deserves a golden value, and promoting all of them is the most common way these records get bloated to the point where nobody trusts any of it.

Three tests decide it. The attribute has to describe the entity rather than an event, so a surname qualifies and a session preference does not. It has to be single valued at a point in time, so a mobile number qualifies and the list of shows somebody has attended does not. And somebody has to consume it outside the system that produced it, since a field nobody reads is a field nobody will notice has gone stale.

Everything failing those tests stays where it was born, on the registration row, joined to the person by the identity link rather than copied onto it. Dietary requirements, badge collection times, the answer to a survey question asked at one show in one year: all of these are facts about a registration and none of them are facts about a person.

The practical effect is a narrow record with maybe a dozen fields, each of which somebody would notice if it were wrong. A wide record with sixty fields, forty of them promoted event answers, is where provenance work goes to die, because the volume makes review impossible and the errors become invisible in the noise. The report your finance lead eventually reads sits on top of the narrow version, and an identity resolution audit trail in J31 covers what that report has to show.

Where this stops

A golden record is a plausible description of a person that no source ever asserted, and two failures follow from that directly.

It can be internally inconsistent. A 2022 first name sitting beside a 2026 job title and a 2023 country describes somebody who existed in that configuration on no particular date. For most reporting that is harmless. For anything time sensitive, such as attributing a registration to the employer the person had at the time, reading the golden record is the wrong move and the source rows are the right one. Point in time questions need the history, and the assembled record is a current view.

It also creates an illusion of completeness. Eight fields populated from four sources looks like a well described contact, and the underlying evidence may be one row with a phone number and three rows with almost nothing. A completeness score computed on the golden record will always flatter the file. Compute it on the source rows as well and publish both, because the gap between them tells you how much of your data quality is real and how much is assembly.

Take one resolved person from your own system this week and try to trace all eight of the fields you report on back to a source row and a date, using only what the pipeline stores today. However long that takes for one record is the cost of every question anyone ever asks about your unified data output, and it is usually the argument that gets provenance funded.

Questions people ask about golden record construction

What is a golden record in identity resolution?
It is the single assembled view of one resolved person or company, built from the values that survived across all the source rows judged to describe that entity. It holds no original data of its own. Every field on it is a copy of a value from a source row, chosen by a rule and traceable back to that row.
What provenance should a golden record store per field?
Four things beside each value: the identifier of the source row it came from, the system that supplied it, the timestamp at which it was observed, and the rule that selected it. Those four turn the question of where a value came from into a lookup instead of an investigation.
Should a golden record be a table or a view?
Derive it from the source rows and the cluster assignment, then materialise the result for query performance. Building it as an editable table invites hand corrections that the next load overwrites, and it makes the record impossible to rebuild when a survivorship rule changes or a merge is reversed.

Related reading

All identity resolution articles