Skip to content

< all problems24 · Level 02, Search

Measure Retrieval with Recall@k and MRR

medium · implement · Evals

Two numbers for retrieval quality.

Recall@k: of the documents that should have been found, what fraction appeared in the top k?

MRR (Mean Reciprocal Rank): 1 / rank of the first relevant hit, 0 if there was none, averaged over all queries. Rank 1 scores 1.0, rank 2 scores 0.5, rank 5 scores 0.2.

Implement recall_at_k(retrieved, relevant, k) and mrr(runs), where runs is a list of (retrieved, relevant) pairs.

  1. A query with no relevant documents returns 0.0, not a nan from dividing by zero.
  2. Ranks are 1-based. The first result is rank 1, not rank 0.
  3. Average over every query, including the ones that found nothing. Skipping failures is how a system scores 0.9 on a benchmark it actually fails.