Automation bias in approvals and why reviewers rubber stamp generated text
Automation bias in approvals is the tendency to accept machine produced content and stop checking it. Two error types follow: omission, where the reviewer misses what the system never flagged, and commission, where the reviewer accepts a wrong suggestion. Measure the edit rate on generated sections against typed ones, then require a per field attestation.
A show director opens the brief for the March edition on a Tuesday afternoon, moves through twelve sections and approves all of them in about four minutes. Eight of those sections arrived prefilled, drafted by a model that read last year's brief and the current registration file. Four were typed by a person that morning. She read the four.
Automation bias in approvals is the name for what happened, and the interesting part is that nobody was careless. The generated sections read well. They were plausible, complete and formatted the same way the good ones are. Under a deadline, a reviewer allocates attention to the places where errors seem likely, and fluent prose does not look like a place where errors are likely.
What the reviewer is actually doing
Approval is a sampling exercise. Nobody re-derives every claim in a thirteen section brief; they check the bits that look risky and let the rest through on the strength of who wrote it and how it reads.
That heuristic worked when every section had an author. You knew that Priya wrote the audience section, you knew Priya is careful about sizing, and the delegation was to a person with a track record. Once a model writes eight of the twelve, the heuristic keeps running but the thing it is delegating to has changed. The reviewer is still asking whether this looks like something a competent colleague would write, and the answer is yes, because producing text that looks like that is exactly what the model does well.
Parasuraman and Riley set out the general case in Human Factors in 1997. They separated use, misuse, disuse and abuse of automation, and defined misuse as over reliance, which they said shows up as failures of monitoring and as decision bias. The word misuse is doing careful work there. The operator has not stopped caring. The operator has reallocated attention on the assumption that the automated component is more reliable than it is.
Why does generated text get less scrutiny than typed text?
Three things stack up, and none of them is a character flaw.
The first is that generated sections arrive complete. A human draft has gaps, a placeholder, a sentence that trails off. Gaps are an invitation to look. A section with no gaps reads as finished, and finished reads as checked.
The second is that the generated sections are usually right. If the model gets the audience segment names correct in nine briefs out of ten, a reviewer who checks the first three and finds nothing wrong has been trained by the evidence to check less on the fourth. That is rational updating on a base rate, and it produces exactly the wrong behaviour on the tenth brief.
The third is volume. A group running eight shows with two editions a year is approving something like two hundred sections a quarter. Attention is the scarce resource in that job, and the reviewer will spend it where the return looks highest.
Two kinds of error, and only one of them is visible
Skitka, Mosier and Burdick published a study in the International Journal of Human-Computer Studies in 1999 using a simulated flight task, comparing people working with a highly but imperfectly reliable automated aid against people working without one. The participants without the aid performed better. What makes the paper useful here is the way they split the failures.
Errors of omission are the ones where the reviewer misses something because the system never raised it. The brief has no non goals section because the template did not generate one and nobody noticed its absence. Nothing is wrong on the page. Something is missing from the page, and a reviewer scanning for wrongness will not find it.
Errors of commission are the ones where the reviewer accepts what the system said even though other information available to them contradicts it. The generated audience section says the target is procurement leads at mid market operators. The show's own registration file from last year says the audience is overwhelmingly owner operators at small sites. Both facts were available. The reviewer approved the first.
Omission errors are the ones that survive. A commission error eventually collides with reality, usually when a salesperson quotes the brief to an exhibitor and the exhibitor disagrees. An omission error just sits there, and the edition ships without a decision anyone made.
Measuring the edit rate, with the arithmetic
You can put a number on this in an afternoon, and the number is more persuasive than the argument.
Tag every section in your brief store by origin: generated or typed. Then, for each approved section, record whether the text changed between the reviewer opening it and the reviewer approving it. Any change counts, including a single word.
Suppose you pull the last forty briefs of a twelve section template, so 480 section reviews. Eight sections per brief arrive generated, which is 320 generated reviews and 160 typed ones. You count 58 generated sections edited before approval and 99 typed ones.
The generated edit rate is 58 divided by 320, or 18.1 per cent. The typed edit rate is 99 divided by 160, or 61.9 per cent. The ratio is about 3.4 to one in favour of typed sections getting touched.
Now the question that matters. Is the generated text three times better than the typed text, or is it getting a third of the attention? You can distinguish those. Take a sample of twenty approved generated sections, have a second person check each one properly against the source data, and count how many contain an error that should have been caught. If that audit finds errors in five of twenty, a 25 per cent defect rate against an 18 per cent edit rate says reviewers are approving faster than the content deserves.
Time from open to approve is a cheaper proxy and it is worth logging anyway. A median of eleven seconds on generated sections against ninety on typed ones tells you most of the story before you audit anything.
A forced attestation, and what it has to ask for
The countermeasure that survives contact with a busy team is small and specific. On any section that arrived generated, approval requires the reviewer to answer one question per field, and the question names the thing they had to look at.
For a sizing claim, the attestation is a source, chosen from a short list: the registration file for a named edition, the exhibitor contract database, an external figure with a link. For a segment name, it is a tick against the segment list the show actually uses. For a date, it is the operations calendar.
Two properties make this work. The attestation is per field rather than per section, so the reviewer cannot discharge eight claims with one click. And it asks for a source rather than a confirmation, so answering it requires having opened something.
Keep the attestation on generated fields only. Applying it to everything converts it into a formality within a month, and you lose the signal that made it useful. This is the same discipline that makes a definition of done checklist for each section work: the check has to be objective and it has to be cheap to fail.
Log the attestation latency. If the median gap between opening a section and completing its attestations is under five seconds, the attestation has become a click and you should say so out loud rather than pretend the control is working.
Does a second approver help?
Adding a second approver does not help. Both reviewers read the same fluent text, both inherit the same assumption, and you have doubled the calendar cost of a section for no change in the failure mode.
Warning banners do not help for long. A line saying this section was generated, please review carefully, is read on the first brief and invisible by the fifth. It carries no information about which claim to check.
Raising the model's quality helps with commission errors and makes omission errors worse, which is the uncomfortable part. The more reliable the generator, the stronger the reviewer's prior that there is nothing to find, and the less likely anyone notices the section that was never generated at all.
The thing that does help, other than attestation, is making the source visible next to the claim. If the audience size in the brief renders with the query and the edition it came from underneath it, checking costs a glance instead of a search. That is a build decision in your brief and publishing workspace rather than a policy decision, and it is worth more than a policy.
Where this stops
Attestation measures whether someone looked. It says nothing about whether they understood what they were looking at, and it cannot catch a reviewer who ticks the registration file box because the registration file is obviously the right source without opening it.
There is also a real cost. A twelve section brief with four attestable fields per generated section adds thirty two decisions to every approval, and on a team publishing three editions a month that is a genuine tax. If your generated content is mostly descriptive, inherited copy that carries no decision, the tax buys you very little. Save the mechanism for the sections that assert something checkable: sizing, segment definitions, dates, prices, capacity.
The other limit is that this only works where the origin of the text is recorded at all. Plenty of workspaces lose that the moment somebody edits a generated field, and then every section looks typed. If your store does not carry an origin flag per field, the measurement above cannot be run and the attestation cannot be targeted, and fixing that comes before any of this. The same applies to fields that arrive prefilled on an intake form, where the person filling in the form is subject to the identical effect one step earlier.
Start here this week. Take the last ten approved briefs, mark each section generated or typed from whatever history you have, and count the edits before approval in each group. If the generated group is being edited at less than half the rate of the typed group, you have the number you need to justify building the attestation, and it took an afternoon rather than a project.
Questions people ask about automation bias in approvals
- What is automation bias in an approval workflow?
- It is the habit of treating machine produced content as already checked. Parasuraman and Riley described this in 1997 as misuse, meaning over reliance that shows up as failures of monitoring and as decision bias. In a brief workflow it looks like a reviewer opening a generated section, seeing well formed prose, and approving it in seconds.
- How do you measure whether reviewers are actually reading generated sections?
- Tag every section by origin, generated or typed, then count how often each kind gets edited before approval. Divide edits by reviews separately for each origin. A large gap in favour of typed sections tells you the generated ones are passing through unread. Time from open to approve is a useful second measure.
- Does adding a second reviewer fix automation bias?
- Not reliably. Both reviewers see the same fluent text and both inherit the same assumption that it came from somewhere sensible. A forced attestation on each generated field, naming the source the reviewer checked against, changes what the reviewer has to do. A second signature only changes how many people did the same thing.
Related reading
- Prefilled form fields make people agree to things they never read
- A definition of done checklist for each section of an event brief