MRR is not "average rank"

An MRR of 0.50 does not mean the correct answer usually sits at position two. It might mean half your queries are perfect and the other half are complete misses. Those are entirely different systems with entirely different fixes, and the metric reports them identically.

That's the main thing worth knowing about MRR, and most explanations bury it.

The metric itself is the simplest one in retrieval. For each query, find the position of the first correct result and take its reciprocal. Rank 1 scores 1.0, rank 2 scores 0.5, rank 3 scores 0.333, rank 10 scores 0.1, and nothing in the top k scores zero. Average across your query set. That's the whole thing: no grades, no logarithms, no normalisation.

The reciprocal is doing something deliberate. Dropping from rank 1 to rank 2 costs you half your score. Dropping from rank 9 to rank 10 costs you about a hundredth. That curve is harsh at the top and nearly flat at the bottom, which is roughly how attention works when someone is looking for one specific thing. In a known-item search, being second genuinely is being wrong for a large fraction of users.

So MRR is the right metric when there's exactly one correct answer and the user stops as soon as they find it. It fits FAQ lookup, entity resolution, autocomplete, and question answering where a single passage contains the answer. It also fits agent tool selection (did the retriever put the correct tool at the top of a list of forty?), which is a use case almost nobody measures and probably should.

It is the wrong metric when the answer has to be assembled from several documents. MRR ignores everything after the first hit, entirely. A system that surfaces one correct chunk and three pieces of garbage scores a perfect 1.0. If your pipeline needs four chunks to compose an answer, MRR will tell you things are fine while users get partial responses. Use nDCG or recall there.

It also gets confused with Hit Rate constantly, and the two are not the same. Hit Rate asks whether anything relevant made the top k and answers 0 or 1. MRR asks how high the first relevant result landed. For a query with four relevant documents where one is found at position 3, Hit Rate says 1.0 and MRR says 0.333.

Reading the number

Since the average is dominated by the extremes, interpret it with something alongside it. We report MRR@10 with two other numbers: the fraction of queries answered at rank 1, and the fraction with no correct answer in the top 10 at all. Three numbers, and the shape becomes obvious in a way the mean alone never is.

The miss rate is the one we watch hardest: a healthy-looking mean hiding a large fraction of complete misses is a worse system than its average suggests. On our own eval set, the three-number view is what tells us where the next week of work should go, in a way the single mean never does.

Broadly: above 0.9 means almost everything lands at rank 1. Between 0.7 and 0.9, usually rank 1 with occasional slips to 2. Between 0.5 and 0.7, a healthy mix with some misses. Below 0.3, retrieval is your bottleneck and no amount of prompt work will help. These bands are rules of thumb from our own corpus, not universal constants; where the cutoffs sit depends on how many relevant documents your queries actually have.

Two ways to get it wrong

The k truncation is invisible in the score. A query whose answer sits at rank 11, measured at k=10, scores zero, identical to a query whose answer isn't in the corpus at all. One of those is a ranking problem and the other is a coverage problem, and MRR cannot tell them apart. Log the raw rank, not just the reciprocal, or you lose the ability to distinguish them later.

And small eval sets swing wildly. With 20 queries, one query moving from rank 1 to rank 3 shifts MRR by 0.033, which is about the size of the improvement most reranker changes produce. If your eval set is small, you are measuring noise and calling it a result. Two hundred queries minimum, and run a paired significance test before declaring anything a winner.

def reciprocal_rank(retrieved_ids, relevant_ids, k):
    relevant = set(relevant_ids)
    for i, doc_id in enumerate(retrieved_ids[:k], start=1):
        if doc_id in relevant:
            return 1.0 / i
    return 0.0

Four lines. The hard part was never the arithmetic. It's knowing which question you're actually asking, and MRR only answers one of them.

The rest of the series: Recall@k on the metric that caps everything downstream, and nDCG@k on measuring order when relevance comes in degrees.