Measuring retrieval quality separately from the answer the model writes

The assistant answers a question about last year's sponsorship terms and gets it wrong. The immediate reaction in every team I have watched is to rewrite the prompt. Somebody adds an instruction about being careful with dates, the answer changes, and everyone moves on. Measuring retrieval quality as its own number is what stops that cycle, because in a large share of these cases the prompt was fine and the passage containing the answer was never in it.

If the evidence never arrives, nothing downstream can recover. That makes the retriever's score a hard ceiling on the system, and a ceiling is worth knowing before you spend a fortnight under it.

The split that makes the score interpretable

Es and colleagues built Ragas around this separation, published in the system demonstrations of the 18th conference of the European Chapter of the Association for Computational Linguistics in 2024, pages 150 to 158. Their framework scores context relevance for the retrieval stage and faithfulness and answer relevance for the generation stage, deliberately as separate dimensions, and does it without ground truth annotations.

The separation is the contribution worth taking, whatever tooling you use. Precision and recall as retrieval measures long predate any of this: Manning, Raghavan and Schütze set them out in Introduction to Information Retrieval, published by Cambridge University Press in 2008, and the definitions have not moved.

What has changed is that the retrieved set now feeds a generator, so recall at a small k matters more than it ever did in classic search. A human searching a corpus scrolls. A prompt takes five passages and stops.

What does recall at five actually tell you?

Build the gold set first. Fifty questions drawn from a real log, each with the source passage that answers it marked by hand. Not the document, the passage, because a 40 page document marked as correct will make almost any retriever look excellent.

Run the retriever over all 50 and record where the marked passage lands. Suppose the results distribute like this: rank 1 for 22 questions, rank 2 for 9, rank 3 for 5, rank 4 for 2, rank 5 for 1, and outside the top five for 11.

Recall at five is 39 divided by 50, which is 0.78.

Mean reciprocal rank adds the information recall throws away. Take one over the rank for each question and average, counting zero for the misses: 22 times 1, plus 9 times 0.5, plus 5 times one third, plus 2 times 0.25, plus 1 times 0.2. That is 22 plus 4.5 plus 1.667 plus 0.5 plus 0.2, or 28.867. Divided by 50 that is 0.577.

Both numbers are worth reporting. Recall at five tells you what fraction of questions could possibly be answered. Mean reciprocal rank tells you whether the correct passage is arriving at the top of the prompt or the bottom, which matters because models read the beginning and end of a context better than the middle. If your recall at five is healthy and your reciprocal rank is poor, the fix is reranking rather than a bigger index.

Splitting the score into retrieval and generation

Now run the whole system on the same 50 questions and grade the final answers. Suppose 31 are correct, so end to end accuracy is 62 per cent.

Two numbers come out of that, and they point in different directions.

The retrieval ceiling is 0.78. You cannot exceed it without improving the retriever, so 22 per cent of your failures are structural.

The conversion rate is 31 divided by 39, which is 0.795. On the questions where the correct passage did reach the prompt, the generator got it right four times in five, and lost the other one.

That decomposition tells you where the next hour goes. Eleven questions failed because retrieval missed. Eight failed with the evidence in hand. If your instinct was to rewrite the prompt, the arithmetic says the prompt is the smaller of the two problems, and it says so with a number you can put in front of anyone who disagrees.

Recompute both after every change. A change that lifts recall at five from 0.78 to 0.86 and leaves conversion at 0.795 predicts end to end accuracy of 0.86 times 0.795, or 68.4 per cent. If the observed figure comes back at 63 per cent, something else moved and you should find out what.

Why five, and when a different k is the honest one

Five is a convention rather than a law, and reporting recall at a k you do not actually use makes the ceiling meaningless.

Measure recall at the number of passages your prompt genuinely carries. If the prompt takes 8, report recall at 8. If a reranker sits between the retriever and the prompt, you need two figures: recall at whatever the first stage returns, say 50, and recall at what survives the rerank. Those two numbers separate a retrieval problem from a ranking problem, and they are frequently different problems with different owners.

The curve between them is informative on its own. Suppose recall at 5 is 0.78, recall at 10 is 0.86, and recall at 50 is 0.88. The retriever is finding almost everything it will ever find by rank 10, so widening the net is nearly exhausted and reranking the top 10 into the top 5 is where the remaining 8 points live. If instead recall at 50 came back at 0.96, the passages are being found and buried, and a reranker is the obvious next build.

Report the k. A recall figure with no k attached is the most common way this measurement gets quietly inflated.

Reading the misses

The 11 misses are the most valuable object in the exercise and they get skipped because reading them is slow.

Sort them into causes. On event archives the same four keep appearing.

• Vocabulary mismatch. The question says "renewal" and the document says "rebooking". The synonym list fixes this and nothing else will.
• The answer spans two passages. A figure in one chunk and the qualifying condition in the next. This is a chunking decision, addressed in how the documents get split, and it is invisible in any aggregate score.
• The document is not in the corpus. No retrieval fix exists. Somebody has to add the file, and the system should be saying it cannot answer rather than reaching for the nearest thing.
• The question is ambiguous. "How many exhibitors last year" across a portfolio of eight shows. The retriever is being blamed for a question that has four correct answers.

Counting those four causes across 11 misses usually produces a distribution nobody predicted, and it redirects the work. Five vocabulary mismatches means write the synonym list this afternoon. Five missing documents means the corpus definition is the problem and no retrieval work will help.

Should you use a reference-free grader?

For monitoring at volume, yes. As the number you steer by, no.

A reference-free context relevance score asks a model whether the retrieved passages look relevant to the question. That is genuinely useful when 640 questions a week arrive and nobody can label them, and it will flag a day when retrieval quality collapsed. It runs unattended and it costs a few pennies a question.

What it cannot do is tell you the retriever missed the passage you know is correct, because it never saw that passage. It scores what came back. A retriever that consistently returns plausible, on-topic, wrong passages scores well on context relevance and fails every question, and that is a common failure on an archive full of near-duplicate sections across eight shows and five years.

Run both. The labelled 50 is the instrument you calibrate against, refreshed when the question mix moves. The automated score is the smoke alarm. Treating the smoke alarm as the instrument is the mistake, and it is easy to make because the automated number is available every day and the labelled one takes an afternoon.

Where this stops

Fifty questions with hand-marked sources is a small sample and the interval around 0.78 is wider than it looks. The standard error is the square root of 0.78 times 0.22 over 50, which is 0.0586, so two standard errors is about 12 points. Recall at five somewhere between 0.66 and 0.90 is what you have actually established. That is enough to compare against a conversion rate of 0.795 and enough to see a large change, and it is not enough to declare a 3 point improvement real.

The deeper limit is that a gold set encodes one person's view of which passage answers each question, and for the questions that matter most, that view is contested. Where two people mark different passages as correct, the disagreement is telling you the question is underspecified, and the honest response is to split it into two questions rather than pick a winner. What people actually ask, and how far it drifts from your fifty, is a separate measurement covered in offline evaluation against live monitoring. Which retrievers to run underneath all of this is the subject of hybrid keyword and vector search, and the pairing of retriever and generator sits in retrieval augmented generation over your own documents. The wider set of controls lives on the AI platform side.

Start by marking the correct source passage for 50 questions your team asked last month. Run the retriever, count how many land in the top five, and put that number next to your end to end accuracy. Whichever gap is larger is the one to work on, and the whole exercise is one afternoon.