Two systems, identical recall, one of them useless

Here are two retrievers answering the same query. Both return the same five documents. Relevance is graded 0 to 3, where 3 fully answers the question and 0 is noise.

Recall@5 is 1.0 for both. Precision@5 is identical. Hit Rate@5 is identical. Every order-blind metric you have says these systems are the same.

They are not the same. If you feed the top three chunks to a language model, System A hands it the two best documents in the corpus and System B hands it two zeros and a marginal. Same recall, opposite outcomes. This is not a contrived example. It is the ordinary failure mode of evaluating a reranked pipeline with metrics that ignore rank.

nDCG@k is the metric that separates them, and it does so by combining two fairly obvious ideas.

Not all relevance is equal. A document that fully answers a question is worth more than one that mentions the topic. So you abandon binary labels and use grades: we use 0 to 3, which is enough resolution to be useful without making judgment calls agonising.

Position matters, with diminishing severity. Rank 1 versus rank 2 is a large difference. Rank 9 versus rank 10 barely registers. A logarithm has exactly that shape, so each document's gain gets divided by log₂(position + 1).

Multiply gain by discount, sum across the top k, and you have DCG:

DCG@k = Σ (2^relᵢ − 1) / log₂(i + 1)   for i = 1 … k

The 2^rel − 1 term is the exponential gain variant, which weights highly relevant documents aggressively. There's a linear variant (rel_i / log₂(i+1)) that's also perfectly valid. Pick one, write it down somewhere, and never quietly switch: comparing an exponential-gain result against a linear-gain one is a mistake that's almost impossible to spot after the fact.

Raw DCG isn't comparable across queries, though. A query with four excellent documents available will naturally score higher than one with a single mediocre match, and that's a fact about your corpus rather than your ranker. So you compute the DCG of the ideal ordering (same judged documents, sorted best first) and divide. That's IDCG, and the ratio is nDCG:

nDCG@k = DCG@k / IDCG@k

Zero to one, comparable across queries, averageable. It answers the only fair question: how close did you get to the best ordering available for this particular query.

Running the numbers

Take a ranking: grades 2, 0, 3, 1, 0 down the top five.

Position 1 contributes (2²−1)/log₂(2) = 3/1 = 3.000. Position 2 contributes nothing. Position 3 gives (2³−1)/log₂(4) = 7/2 = 3.500. Position 4 gives 1/2.322 = 0.431. Position 5, nothing. DCG@5 = 6.931.

The ideal ordering of those same documents is 3, 2, 1, 0, 0, which gives 7 + 3/1.585 + 1/2 = 7 + 1.893 + 0.500 = 9.393.

nDCG@5 = 6.931 / 9.393 = 0.738.

Read that as: this ranking captured roughly 74% of the ranking value that was actually available. The entire shortfall comes from the best document sitting at position 3 instead of position 1.

Now put a reranker in front of it that produces the ideal order. DCG becomes 9.393 and nDCG hits 1.0. Same documents, same recall, and a 26-point quality improvement that Recall@5 would have reported as exactly zero change. That is the business case for a cross-encoder in a single number.

Back to the two systems at the top: System A scores 1.000, System B scores 0.479. Recall called them equivalent. nDCG says one is twice as good as the other, which matches what anyone reading the outputs would tell you.

import numpy as np

def dcg_at_k(grades, k):
    g = np.asarray(grades, dtype=float)[:k]
    return float(np.sum(((2 ** g) - 1) / np.log2(np.arange(2, len(g) + 2))))

def ndcg_at_k(ranked_grades, all_judged_grades, k):
    idcg = dcg_at_k(sorted(all_judged_grades, reverse=True), k)
    return dcg_at_k(ranked_grades, k) / idcg if idcg > 0 else 0.0

ndcg_at_k([2, 0, 3, 1, 0], [3, 2, 1, 0, 0], k=5)   # 0.738

Choosing k, and the mistakes it invites

Set k to however many results the consumer actually consumes, not how many you retrieve. If you're feeding three chunks to a model, nDCG@3 is your number. nDCG@10 is measuring seven documents nobody will ever read.

On the same ranking above, nDCG@3 is 0.692 and nDCG@5 is 0.738. Not a huge gap here, but on a pipeline where the reranker is doing real work the two can diverge substantially, and reporting the wrong one will make a bad system look acceptable.

Report nDCG@3 alongside nDCG@10: the first for context-window quality, the second for candidate generation. The two can tell different stories, and the gap between them is where a reranker earns its keep.

Four other things that bite you, or nearly do.

Unjudged documents score zero, which sounds harmless until you're comparing a new model against a judgment pool built from the old one. The new model surfaces genuinely good documents nobody judged, they count as irrelevant, and the better system loses. Pool judgments across every system you're comparing, or accept a bias you can't quantify.

IDCG has to come from the full judged set, not from what you returned. If you sort only your own results, a system that missed the best document gets a flattering ceiling and an inflated score. This is an easy bug to write and a hard one to notice.

nDCG is normalised within a query, not across corpora, so "our nDCG is 0.82 and the paper reports 0.71" means nothing unless it's the same benchmark on the same data. It's an easy comparison to make by accident.

And the mean hides everything. An average nDCG of 0.78 is perfectly compatible with a long tail of queries scoring under 0.2. Ship the distribution: p10, median, and the twenty worst queries. That tail is a bug list, not a statistic.

The rest of the series: Recall@k covers the metric that caps how good nDCG can possibly get, and MRR@k covers what to use when there's exactly one right answer.