Revisiting RAG Retrievers: An Information Theoretic Benchmark

TL;DR

This paper introduces MIGRASCOPE, an information-theoretic framework for systematically analyzing and comparing RAG retrievers, revealing insights on their redundancy, synergy, and how ensembles can outperform individual retrievers.

Contribution

It develops principled metrics based on information theory to evaluate retrievers and demonstrates that ensembles of retrievers can surpass single methods in RAG systems.

Findings

Ensemble of retrievers outperforms individual retrievers.

New metrics quantify retrieval quality, redundancy, and synergy.

Insights into the structure and contribution of state-of-the-art retrievers.

Abstract

Retrieval-Augmented Generation (RAG) systems rely critically on the retriever module to surface relevant context for large language models. Although numerous retrievers have recently been proposed, each built on different ranking principles such as lexical matching, dense embeddings, or graph citations, there remains a lack of systematic understanding of how these mechanisms differ and overlap. Existing benchmarks primarily compare entire RAG pipelines or introduce new datasets, providing little guidance on selecting or combining retrievers themselves. Those that do compare retrievers directly use a limited set of evaluation tools which fail to capture complementary and overlapping strengths. This work presents MIGRASCOPE, a Mutual Information based RAG Retriever Analysis Scope. We revisit state-of-the-art retrievers and introduce principled metrics grounded in information and statistical estimation theory to quantify retrieval quality, redundancy, synergy, and marginal contribution. We further show that if chosen carefully, an ensemble of retrievers outperforms any single retriever. We leverage the developed tools over major RAG corpora to provide unique insights on contribution levels of the state-of-the-art retrievers. Our findings provide a fresh perspective on the structure of modern retrieval techniques and actionable guidance for designing robust and efficient RAG systems.