How far to trust LLM as judge reliability when grading your assistant

Show week closes and the assistant over your event warehouse has answered 640 questions. Somebody on the exec team asks the obvious thing: were they right? Nobody is going to read 640 answers, so the plan becomes what it always becomes, which is to get a second model to mark the first one. That plan is reasonable, and the question of llm as judge reliability is the one that decides whether the resulting number belongs anywhere near a board pack.

The short version is that automated graders work and are biased at the same time, and both halves of that sentence matter.

What the judge is actually being asked

Grading is not one task. Asking a model whether an answer is factually supported by a set of aggregates is a narrow, checkable job. Asking it which of two answers a show director would prefer is a taste judgement with no ground truth at all. Most teams build the second and report it as though it were the first.

Zheng and colleagues, in the MT-Bench and Chatbot Arena paper at the NeurIPS 2023 Datasets and Benchmarks track, tested exactly the preference case. Their headline is that strong judges "can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans". That is a genuinely useful result and it is the reason automated grading is worth running.

The same paper names the failure modes: position bias, verbosity bias, self-enhancement bias, and limited reasoning ability. Those are not caveats bolted on at the end. They are the paper's own account of what breaks.

For an event assistant the narrow job is usually the one worth automating. A question like "how many exhibitors from the 2025 edition have rebooked" has a right answer sitting in a table, and the grading task is to check whether the prose matches the aggregates the model was handed. That reduces to a comparison, and a model does comparisons well. The taste question, whether the answer was framed usefully for a show director four weeks out, is real and worth asking, and it has no ground truth you can hold the grader to.

Write the rubric for the narrow job first. If you cannot state what would make an answer wrong, an automated judge will invent a standard and apply it consistently, and consistency will get mistaken for correctness.

How badly does position bias distort a grade?

Badly enough to reverse a result. Wang and colleagues showed in 2023 that the ranking of two candidate answers "can be easily hacked by simply altering their order of appearance in the context", and in their experiment Vicuna-13B was made to beat ChatGPT on 66 of 80 queries with ChatGPT itself acting as judge, purely through ordering.

Sixty-six out of eighty is 82.5 per cent, and the whole effect came from which answer went first. If you are comparing your new prompt against last month's prompt and you always put the new one second, you have built a measurement that will lie to you in a consistent direction, which is the worst kind.

The mitigation is cheap. Run every pairwise comparison twice with the order swapped and count only the pairs where both runs agree. Pairs that flip go to a human or get dropped. You lose some throughput and you gain a number that survives being questioned.

Verbosity bias has a similar shape and a similar fix. If the judge prefers longer answers, then a prompt change that makes answers longer will look like a quality improvement. Hold answer length roughly constant across the arms of any comparison, or record length alongside the grade so you can see whether the two moved together.

Eighty per cent agreement can mean almost nothing

This is the part that gets skipped, and it is arithmetic rather than opinion.

Suppose your assistant is good. The model judge marks 34 of 40 answers as passing, so its pass rate is 85 per cent. A human regrades the same answers and also passes 85 per cent of them. They agree on 80 per cent of the items.

Work out how much agreement you would get from two graders who were simply guessing at those rates. Both say pass: 0.85 times 0.85, which is 0.7225. Both say fail: 0.15 times 0.15, which is 0.0225. Expected agreement by chance is 0.745.

Chance-corrected agreement is the observed agreement minus the chance agreement, divided by one minus the chance agreement. That is 0.80 minus 0.745, over 1 minus 0.745, which is 0.055 divided by 0.255, or 0.216.

So an 80 per cent agreement rate on a task where 85 per cent of answers pass has almost no information in it. The judge is doing barely better than a coin weighted to say pass. On a harder question set where the pass rate is 50 per cent, chance agreement drops to 0.5 and the same 80 per cent observed agreement gives 0.60, which is a genuinely useful grader.

The practical consequence: never report judge agreement without reporting the pass rate next to it, and be suspicious of any grading exercise where nearly everything passes.

A grading pass you can run on forty answers

Take a show week's log and pull 40 answers at random. Have the model grade each one against a written rubric with three outcomes: supported by the aggregates provided, contradicted by them, or unanswerable from them. Three outcomes beats a one-to-five score, because a one-to-five score invites the judge to average away a factual error.

Then have a person regrade a sample. Here is where most teams under-invest, and the arithmetic is unforgiving.

If a human regrades 10 answers and disagrees with the judge on 2, the point estimate for the disagreement rate is 20 per cent. The standard error is the square root of 0.2 times 0.8 divided by 10, which is the square root of 0.016, or 0.126. Two standard errors is 0.25. Your interval runs from roughly zero to 45 per cent, which is compatible with a nearly perfect judge and with a nearly useless one.

To pull that interval down to plus or minus 10 points you need two times the square root of 0.16 divided by n to equal 0.10. Square both sides: 0.64 divided by n equals 0.01, so n is 64. Sixty-four human regrades is a couple of hours for someone who knows the data. Ten is theatre.

There is a shortcut that buys back some of that cost. Stratify. Regrade every answer the judge marked as contradicted, since those are rare and each one is informative, and take a random sample of the ones it passed. Report the two strata separately instead of blending them into a single percentage.

The stratified version also tells you which direction the judge errs in, which the blended figure hides. Run the judge across the whole 640 and suppose it flags 38 as contradicted. A human reads all 38 and agrees with 25 of them. Thirteen false alarms out of 38 is a 34 per cent false positive rate on the flagged set, annoying and fairly harmless, because a person was going to read those anyway. Then take 64 of the 602 it passed and regrade those. If 5 of the 64 should have failed, that is 7.8 per cent of the passed set slipping through, which across 602 answers implies roughly 47 wrong answers nobody flagged. The two error rates cost you completely different things and there is no reason to average them into one figure.

Should the judge ever see the arithmetic?

No, and this is where I part company with most of the tooling.

The limited reasoning ability that Zheng and colleagues named is not evenly distributed. A judge asked whether an answer's prose is consistent with a list of named aggregates does fine. A judge asked whether 2,702 divided by 14,382 is correctly reported as 18.8 per cent is being asked to do the thing language models are worst at, and it will produce a confident verdict either way.

Check numeric claims with code. Parse the figures out of the answer, recompute them from the same aggregates the answer was given, and compare with a tolerance. That check is deterministic, costs nothing, and catches a class of error the judge will pass. Leave the model to grade tone, completeness, hedging and whether the answer actually addressed the question. The division between the two jobs is worth drawing explicitly, and it is the same division that makes computing in SQL and passing the result in the right default for the assistant itself.

What to store with every grading run

A grade with no provenance is a number nobody can re-derive in three months.

• The rubric text, verbatim, as it was at the time. Rubrics drift and a drifted rubric makes two runs incomparable.
• The judge model and version string, because a silent upgrade changes the grader. Holding an eval set stable across a model swap is its own discipline, covered in regression testing when the model underneath changes.
• The pass rate, so anyone reading the agreement figure can compute the chance-corrected version.
• The order each pair was presented in, and the result of the swapped run.
• Which items a human regraded, and what they said.

One thing does not belong in the judge's remit at all, and it is worth separating explicitly. Where the answer depended on retrieved documents, a low grade can mean the model wrote badly or that the right passage never arrived, and the judge cannot tell those apart from the output alone. Score the retriever on its own, using the method in measuring retrieval quality separately, and hand the judge only the answers where the evidence was present.

Where this stops

An automated judge measures agreement with a rubric. It does not measure whether the rubric is the right one, and on event data the rubric is usually where the real disagreement lives.

Two experienced people will genuinely differ on whether an answer that reports verified attendance without flagging 60 per cent scan coverage is correct. The model will confidently pick a side, and it will pick the same side every time, which reads like consistency and is actually a frozen opinion nobody signed off.

The second limit is coverage. A judge grades the questions you sampled. If your sample comes from the eval set rather than the log, you are grading the questions you thought of, and the difference between those two populations is the subject of offline evaluation against live monitoring. Neither exercise substitutes for the other, and the wider set of controls sits on the AI platform side.

Start this week by pulling 40 answers from your last show and grading them by model, then regrading 64 of your accumulated log by hand over two sittings. Put the pass rate and the chance-corrected agreement on the same line. If the corrected figure comes in under about 0.4, your judge is not yet measuring anything you can act on, and the rubric is the first thing to rewrite.