A brief quality score you can apply before work starts

In the quarterly portfolio review somebody asks why two of the eight shows always need a rebrief around week ten. Everybody in the room knows the answer and everybody gives the same version of it, which is that those two shows write bad briefs. Nobody can evidence it. The production lead has a feeling, the studio manager has three examples, and the two show directors involved have a different explanation entirely, which is that the studio starts before the brief is finished.

The argument repeats every quarter because there is no brief quality score. Introduce one and the argument happens once, on numbers, and then stops.

Five items, scored zero to two

Keep the rubric small enough that scoring a brief takes four minutes. Five items, each worth zero, one or two, for a total out of ten.

A measurable objective. Two if the brief names a number and a direction, such as 1,400 pre registered buyers from the retrofit segment. One if it names a direction with no number, such as grow the buyer audience. Zero if it names an activity, such as run an integrated campaign.

A named audience. Two if a photographer could act on it: job title, buying situation, and what that person is trying to solve. One if it names a segment with no situation. Zero if it lists everyone who might come.

A stated constraint. Two if the brief names a budget, a channel restriction, a legal mandatory or a fixed date that genuinely rules something out. One if a constraint is implied. Zero if the brief reads as though anything is possible.

One accountable owner. Two if a single person is named for the brief as a whole. One if a team or a role is named. Zero if the approver field holds a distribution list.

A dated deadline. Two if there is a date on the decision, distinct from the date of the edition. One if only the edition date appears. Zero if no date is present.

Four of those five track criteria Doran set out in Management Review in 1981 for writing a usable objective, where the ones people drop are assignable and time related. Those are precisely the items that score zero most often in event briefs, which is worth sitting with for a moment.

Scoring twenty briefs across a portfolio

Here is the exercise on twenty briefs drawn from eight shows over four quarters, scored by one production lead who wrote none of them.

The totals come out as three briefs at 9, six at 8, four at 5, four at 4 and three at 3. That is 27 plus 48 plus 20 plus 16 plus 9, which is 120 points across 20 briefs, so the mean is 6.0 out of ten.

The mean is the least interesting number in that paragraph. Sort the twenty scores and the tenth and eleventh are both 5, so the median is 5, well below the mean, and the distribution has two clumps: nine briefs at 8 or 9, and eleven at 5 or below. There is almost nothing in the middle. A portfolio with a mean of 6.0 sounds like a portfolio with a consistent mild problem. This one has two populations, and any intervention aimed at the average will miss both.

Now break the mean into its five items. Measurable objective averages 0.9. Named audience 1.5. Stated constraint 1.4. Accountable owner 0.7. Dated deadline 1.5. Those sum to 6.0, and they say something the total cannot: audiences and deadlines are mostly fine, and the portfolio is systematically failing to name a number and a person.

That is a two hour fix at template level and it moves the portfolio mean from 6.0 to about 8.4 without anybody writing better prose. Whether the underlying briefs get better is a separate question, taken up below.

Do two people scoring the same brief agree?

Worth checking before anyone reports the number upward, because a rubric that two competent people apply differently is a measurement of the scorer.

Have two people score the same twenty briefs on the item that looks most subjective, the accountable owner. Suppose they agree on 16 of 20, so observed agreement is 0.80. Chance agreement has to come out of it. If both scorers hand out zero about 60 per cent of the time, one about 25 per cent and two about 15 per cent, the agreement you would expect from chance alone is 0.6 times 0.6 plus 0.25 times 0.25 plus 0.15 times 0.15, which is 0.36 plus 0.0625 plus 0.0225, or 0.445.

Cohen's kappa, from his 1960 paper in Educational and Psychological Measurement, is the observed agreement minus the chance agreement, divided by one minus the chance agreement. Here that is 0.355 over 0.555, which is 0.64.

A kappa of 0.64 is usable for a rubric that drives a conversation and too weak to drive a gate that blocks production. The repair is a worked definition rather than a better scorer: write down what a two looks like for each item, with one real example from your own briefs, and re run the exercise. Agreement on this item usually rises fast, because most of the disagreement turns out to be two people holding different views on whether a named role counts as a named person.

What does a score of six out of ten actually predict?

Rework, and it predicts where rather than how much.

An item scoring zero is a decision that has not been made, and an unmade decision does not disappear. It surfaces later, at the point where somebody has to act on it, and by then there is work to throw away. A brief with no measurable objective produces a campaign that cannot be evaluated, which costs an argument in November. A brief with no accountable owner produces a six week comment thread, because there is nobody whose job it is to close it.

The BetterBriefs Project, which surveyed over 1,700 marketers and agency staff across more than 70 countries and published with the IPA in 2021, found 69 per cent of marketers and 73 per cent of agencies saying rebriefs happen too often, with time and money lost each time. The same study put respondents' estimate of budget wasted through poor briefing at 33 per cent.

Scoring before work starts moves the cost of a change from a fortnight to an hour. It does not remove the change, and a team that scores briefs expecting rebriefs to stop will be disappointed. What stops is the surprise.

Publishing the distribution

Score in private and the number becomes a stick. Publish the distribution across the portfolio and it becomes a comparison, which is the only form in which anyone acts on it.

Publish four things: the mean, the median, the five item averages, and the count of briefs scoring below the gate. Do not publish a league table by show director, because the first thing that produces is inflation in the scores rather than in the briefs. Publish the score beside the completion figure too, since the two will disagree constantly and the arithmetic behind that disagreement is worth understanding before anyone reports either number upward. Show the item averages by show if you must show anything by show, since an item average points at a template or a habit and a total points at a person.

The distribution also tells you whether the score is worth keeping. If every brief lands between 7 and 9 within two quarters, the rubric has stopped discriminating and the gate has become a formality, which is a decent time to raise the bar on what counts as a two. Tracking that over a year needs the scores stored beside the briefs themselves, which is one of the things a brief and review workspace is for, and one reason a score kept in a spreadsheet tends to survive about two quarters.

Where the score misleads

The rubric measures whether a decision was recorded. It has nothing to say about whether the decision was any good.

A brief can score ten out of ten and point an edition at an audience that will not come. The objective is measurable, the audience is named, the constraint is stated, one person owns it, the deadline is dated, and the whole thing is wrong. The score will be a perfect ten and the campaign will fail, and the score will have been honest all along about what it was measuring, which is the presence of decisions and nothing else. Separating those two questions is the point of scoring section by section against a rejected alternative, and even that only tests whether a choice existed.

The second limit is that five items cannot cover a twelve section template. Non goals are absent from this rubric, and they are one of the sections most worth having, because they prevent mid cycle requests. Adding a sixth item is tempting and it lengthens the scoring past the four minutes that keeps the habit alive. I would rather keep five and handle non goals through the decision and description split in the template itself, where the section is either present or it is not, and no scoring is required.

Take your last five signed briefs this week and score them on the five items. Twenty minutes. If the item averages look anything like 0.9 for the objective and 0.7 for the owner, you already know which two fields to change in the template before you change anything else.