What belongs in an integration runbook for an event tech stack
An integration runbook holds one page per connection: the owner, the credential and its expiry date, the schedule, the expected row count band, the known failure signatures and the vendor escalation contact. Its test is whether someone who did not build the integration can restore it from the page alone.
At 02:00 the registration feed stops. The person on call has never seen this integration, does not know which credential it uses, cannot tell whether 0 rows is a failure or a quiet night, and has no idea whether waking the data lead is proportionate.
An integration runbook is the page that answers those four questions in under two minutes. Most event teams have documentation, in the sense that there are Confluence pages describing what the integrations do. Very few have the page you would actually want at 02:00, and the difference between the two is worth being precise about.
The 02:00 test
Every candidate item for the runbook gets the same question asked of it. Would somebody who has never seen this system need it at two in the morning?
Architecture diagrams fail that test. So do descriptions of business purpose, lists of stakeholders, and the history of why the integration was built. All of that belongs somewhere, and somewhere is not this page.
What passes: who owns it, what it authenticates with, when it runs, what normal looks like, what the failures look like, and who to call. Six items, one page, one page per connection.
The case for writing them is stronger than most operational documentation. Google's site reliability engineering book states that "thinking through and recording the best practices ahead of time in a 'playbook' produces roughly a 3x improvement in MTTR as compared to the strategy of 'winging it'" (Beyer and colleagues, 2016). Apply that ratio to an event stack and the arithmetic is easy: a 90 minute incident becomes a 30 minute one, and if you have a dozen integration incidents a year, the pages pay for themselves in the first quarter. The ratio comes from Google's environment rather than yours, so treat it as an argument for the practice instead of a forecast, and measure your own once you have a few incidents to compare.
The six things, one integration at a time
The owner, by name. A team name is a way of saying nobody. Put a person on the page and a deputy under them, and accept that this makes the page go stale when people move, because a page that goes visibly stale is better than one that was never true.
The credential and its expiry. Which credential, what type, where it is stored, when it expires, and who can reissue it. Expiry dates belong in a calendar as well as on the page. Rotating the thing without dropping a feed is a procedure of its own, and credential rotation covers it.
The schedule and the expected duration. When it runs, how long it normally takes, and what the longest acceptable run is. Duration is the field people leave out and the one that catches slow degradation.
The expected row count band. Covered below, because it is the item that turns a runbook from documentation into monitoring.
The failure signatures. Every error you have already seen, what it meant, and what fixed it. Two or three entries at the start, growing every time something breaks. This section is the runbook's real value and it can only be written by the people who were there.
The vendor contact. Support portal URL, account reference, the support tier you actually pay for, and the response time it promises. Also whether support is available at 02:00 in your timezone, which is frequently the whole answer to whether you should escalate now or at eight.
Keep the page in the same repository as the integration code, in plain markdown, one file per connection. Wikis drift because editing them is a separate act from changing the thing they describe; a file that sits next to the code gets updated in the same commit as the change, and a reviewer can see when it did not. It also survives the platform migration that eventually eats every wiki.
Format matters less than length. If the page runs past a screen and a half, the useful parts are buried and the reader at 02:00 will skim past them. Long explanations belong in a linked document that nobody needs during an incident.
Why an expected row count beats an alert on failure
Alerting on failure catches the loud problems. Almost every integration incident I have had to explain afterwards was quiet.
A feed that returns 200 with an empty array. A pull that succeeded against a filter that silently matched nothing. A job that ran, wrote 40 rows instead of 900, and exited zero. None of those fire a failure alert, and all of them are visible instantly against an expected row count.
Set the band from history rather than from opinion. Take the median of the last 14 runs on the same weekday, since event data is strongly weekly, and alert outside a multiplier of it. If the median is 950 rows, a floor at 40 per cent gives 380 and a ceiling at 250 per cent gives 2,375. Those are wide, deliberately, because a launch day genuinely produces four times a normal day and an alert that fires on every good day gets muted within a fortnight.
Write both numbers on the page next to the median, and write the date you calculated them. A band computed in February for a show that opens registration in June is wrong by June, and a reader can only tell if you dated it.
The band also gives the person on call the answer to the hardest 02:00 question. Zero rows against a floor of 380 is an incident. Zero rows against a floor of 0, on a feed that legitimately has nothing to send most nights, is a Tuesday.
What makes a runbook stay true?
Two mechanisms, and neither is discipline.
The first is to put the page where the alert points. The alert that fires at 02:00 should carry a link to the runbook page for that specific integration. That single change is worth more than any amount of encouragement to keep documentation current, because the page is now read at the exact moment somebody has both the motivation and the knowledge to fix it.
The second is to require an edit as part of incident closure. Not a post mortem, which is heavier than most event teams will sustain, just one line added to the failure signatures section. If an incident closes without a runbook edit, either the failure was already documented, which is fine and worth noting, or the page just fell behind.
Numbers on the page need their own refresh trigger. Tie the row count recalculation to the show cycle: recompute at registration launch, at the early bird deadline, and after close. Three times a year per show, from a query you write once.
Some of what the runbook records should come from the vendor's own response headers, and it is worth knowing which of those you can trust. Microsoft publishes rate limit headers for Dataverse and says directly that clients should not depend on the values to control how many requests they send, because they are intended for debugging purposes (Microsoft, 2026). Read them into your logs for diagnosis, and drive the retry behaviour from the responses you actually receive.
How do you test a runbook?
Hand it to somebody who did not build the integration and ask them to restore it.
Pick a quiet week. Break the integration deliberately, by revoking a credential or pointing it at a wrong endpoint, and have the tester recover it using the page alone, with no access to the person who wrote it. Every question they have to ask out loud is a gap, and the transcript of their questions is your edit list.
This costs an afternoon per integration and it is the only test that works, because the author of a runbook cannot see what they left out. They know the thing they forgot to write down; that is precisely why they forgot to write it down.
Date the page when it passes. A runbook with a test date three months old reads very differently from one with no date at all, and the reader at 02:00 needs to know which they are holding.
Do the test outside the freeze period, since deliberately breaking a feed is exactly the kind of change a show week freeze exists to prevent.
Where this stops
A runbook is a record of the failures you have already had. It is silent about the ones you have not, and the first occurrence of anything new gets handled the hard way whatever is on the page.
There is also a real failure mode where the runbook becomes the reason nobody fixes the underlying problem. A well documented manual recovery for a job that fails every fortnight is a solved problem in the wrong direction. Watch the failure signatures section: an entry that keeps getting used is a candidate for automation, and the page should carry a note saying so rather than quietly making the manual fix sustainable.
The last limit is scope. A runbook per integration says nothing about the order to restore them in when several fail at once, which is a different document and only worth writing if your event stack has enough interdependence to need it.
This week, pick the integration you would least like to lose during show week and write its six items on one page. If you cannot fill in the credential expiry date without asking somebody, that gap is the first thing the page has already earned.
Questions people ask about integration runbook
- What should an integration runbook contain?
- Six things per connection: a named owner, the credential in use with its expiry date and where it is stored, the schedule and expected duration, the normal row count range, the failure signatures you have already seen with what each one means, and who to contact at the vendor with the account reference. Everything else is optional.
- Is a runbook worth the effort for a small event team?
- Google's site reliability engineering book reports that recording best practices in a playbook ahead of time produces roughly a 3x improvement in mean time to repair compared with improvising. A small team feels that more sharply than a large one, because a small team usually has exactly one person who knows how each connection works.
- How do you know a runbook is still accurate?
- Have somebody who did not build the integration restore it using only the page. Anything they have to ask about is a gap, and the questions they ask are the edit list. Run this once a year, before the busy season, and date the page when it passes so a reader can tell a current page from an abandoned one.