Measure Retrieval with Recall@k and MRR
Two numbers for retrieval quality.
Recall@k: of the documents that should have been found, what fraction appeared in the top k?
MRR (Mean Reciprocal Rank): 1 / rank of the first relevant hit, 0 if there was none, averaged over all queries. Rank 1 scores 1.0, rank 2 scores 0.5, rank 5 scores 0.2.
Implement recall_at_k(retrieved, relevant, k) and mrr(runs), where runs is a list of (retrieved, relevant) pairs.
- A query with no relevant documents returns
0.0, not ananfrom dividing by zero. - Ranks are 1-based. The first result is rank 1, not rank 0.
- Average over every query, including the ones that found nothing. Skipping failures is how a system scores 0.9 on a benchmark it actually fails.