Recall@k is the ceiling on everything else

Most teams treat Recall@k as the metric you outgrow. You start there, you discover nDCG, you move on. That instinct is backwards, and following it is a good way to burn time tuning a part of the pipeline that was never the problem.

The setup is ordinary. Vector search returning 50 candidates, a cross-encoder reranking them, the top 5 going into the model's context. Answers come back incomplete, not wrong exactly, just missing pieces that are obviously in the corpus. So the usual response kicks in: swap rerankers, tune the prompt, argue about chunk size.

The actual problem is often upstream. A share of the relevant documents were never in the candidate set to begin with. The reranker is doing its job perfectly. It just never sees them.

That's the case for Recall@k, and it's stronger than it first appears. A reranker cannot promote a document that retrieval never returned. A model cannot cite a chunk it never received. Whatever your first stage misses is gone, permanently, before any of the interesting machinery gets a turn. Recall@50 isn't one metric among several. It's the ceiling on nDCG@5, on MRR, on answer quality, on everything downstream.

Which makes it the first thing to measure, not the thing you graduate from.

What it measures

Count the relevant documents that appear anywhere in your top k. Divide by the total number of relevant documents that exist. Position doesn't matter: rank 1 and rank 47 count identically.

That order-blindness is what people object to, and at the candidate-generation stage it's exactly right. You don't care where a document sits in a 50-item list that's about to be reranked. You care whether it's there at all.

Take a query with four relevant documents in the corpus, where your retriever surfaces three of them at positions 2, 5 and 9. Recall@3 is 0.25, Recall@5 is 0.50, Recall@10 is 0.75. Push k to 100 and it's still 0.75, because the fourth document never appears at any depth.

Two things fall out of that. Recall never decreases as k grows, which is why "our Recall@1000 is 0.98" tells you almost nothing. And more usefully, it plateaus. When widening k stops buying you anything, the missing documents aren't ranked low, they're invisible: a vocabulary mismatch, a chunk boundary that split the answer, an embedding that puts the document somewhere the query will never reach. Plotting Recall@k against k and finding the elbow is the cheapest diagnostic in retrieval, and it's the one that tells you whether to work on your retriever or your ranker.

The trap that invalidates most published recall numbers

Recall is capped by k whenever a query has more relevant documents than k.

A query with 20 relevant documents, measured at k=10, has a maximum achievable score of 0.50. A flawless retriever scores 0.50. Now average across an eval set where some queries have two relevant documents and others have thirty, and your mean Recall@10 is largely a statement about the composition of your eval set rather than the quality of your system.

This is the reason to report the achievable ceiling alongside the score. A recall of 0.62 against a ceiling of 0.71 is a very different result from 0.62 against a ceiling of 1.0, and reporting only the first number is close to dishonest even when it's unintentional.

There's a related decision nobody documents: macro versus micro averaging. Macro-averaging takes the mean of per-query recall and weights every query equally. Micro-averaging divides total relevant-found by total relevant across the whole set, which lets queries with many relevant documents dominate the result. For retrieval evaluation macro is almost always what you want. The two can differ by a lot, so say which one you used.

Where the denominator comes from, and why it's a problem

Recall has a dependency the other metrics don't. nDCG and MRR only need judgments on what you returned. Recall needs to know how many relevant documents exist, which for any real corpus you cannot determine exhaustively.

So in practice the denominator comes from pooling: union the top-k from every system you're evaluating, judge that pool, treat it as the universe.

The consequence catches people out. Your recall numbers are relative to the pool, not to truth. A genuinely better retriever that surfaces relevant documents no previous system found will have those counted as irrelevant, and score worse than the baseline it beats. If you're evaluating a new model against an old judgment set, you are systematically penalising it for being novel. Re-pool and re-judge, or accept that your comparison is biased in a direction you can't measure.

There's no clean answer for how much pooling bias distorts a given set of numbers. It's the part of retrieval evaluation worth trusting least.

What to actually do

Measure Recall@50 before you touch anything else. If it's low, the reranker work you were planning is not the highest-value thing available to you. No amount of ordering improvement recovers documents that never entered the pipeline.

Pair it with nDCG@5 for context-window ordering and MRR@10 for top-of-list precision. Three numbers, three distinct failure modes, minimal overlap between them. Recall diagnoses candidate generation, the other two diagnose ranking, and doing them in that order keeps you from optimising a stage that was never the bottleneck.

One thing worth stating plainly: exclude queries with zero relevant documents from the average rather than scoring them zero. Scoring them zero drags the mean down silently and hides real performance, and it's an easy bug to ship because the code looks correct.

def recall_at_k(retrieved_ids, relevant_ids, k):
    relevant = set(relevant_ids)
    if not relevant:
        return None                       # exclude, don't score as 0
    return len(relevant & set(retrieved_ids[:k])) / len(relevant)

def achievable_recall_at_k(runs, k):
    """The ceiling, given queries with more relevant docs than k."""
    caps = [min(k, len(rel)) / len(rel) for _, rel in runs if rel]
    return sum(caps) / len(caps) if caps else 0.0

Report both. The second number is the one that keeps the first honest.

Next in this series: nDCG@k, for when ordering within the top k is what matters, and MRR@k, for when there's exactly one right answer.