RARE: Redundancy-Aware Retrieval Evaluation Framework for High-Similarity Corpora

TL;DR

RARE is a new evaluation framework that accounts for redundancy in corpora, improving the assessment of retrieval systems in real-world, high-similarity document collections.

Contribution

It introduces a method to decompose documents into atomic facts and enhance data generation with CRRF, creating more realistic benchmarks for redundant corpora.

Findings

Applying RARE to Finance, Legal, and Patent data shows significant drops in retriever performance, highlighting robustness gaps.

RARE enables domain-specific RAG evaluations that better reflect real-world conditions.

Abstract

Existing QA benchmarks typically assume distinct documents with minimal overlap, yet real-world retrieval-augmented generation (RAG) systems operate on corpora such as financial reports, legal codes, and patents, where information is highly redundant and documents exhibit strong inter-document similarity. This mismatch undermines evaluation validity: retrievers can be unfairly undervalued even when they retrieve documents that provide sufficient evidence, because redundancy across documents is not accounted for in evaluation. On the other hand, retrievers that perform well on standard benchmarks often generalize poorly to real-world corpora with highly similar and redundant documents. We present RARE (Redundancy-Aware Retrieval Evaluation), a framework for constructing realistic benchmarks by (i) decomposing documents into atomic facts to enable precise redundancy tracking and (ii) enhancing LLM-based data generation with CRRF. RAG benchmark data usually requires multiple quality criteria, but LLMs often yield trivial outputs. CRRF scores criteria separately and fuses decisions by rank, improving the reliability of generated data. Applying RARE to Finance, Legal, and Patent corpora, we introduce RedQA, where a strong retriever baseline drops from 66.4% PerfRecall@10 on 4-hop General-Wiki to 5.0-27.9% PerfRecall@10 at 4-hop depth, revealing robustness gaps that current benchmarks fail to capture. RARE enables practitioners to build domain-specific RAG evaluations that faithfully reflect real-world deployment conditions.