CoverageBench: Evaluating Information Coverage across Tasks and Domains
Saron Samuel, Andrew Yates, Dawn Lawrie, Ian Soboroff, Trevor Adriaanse, Benjamin Van Durme, Eugene Yang

TL;DR
CoverageBench introduces a unified evaluation suite for measuring information coverage across diverse tasks and domains, addressing limitations of traditional relevance metrics in retrieval systems, especially within retrieval-augmented generation.
Contribution
The paper develops a comprehensive testbed for evaluating information coverage, integrating multiple collections and providing standardized datasets for research.
Findings
Provides a suite of collections for coverage evaluation
Addresses limitations of traditional relevance metrics
Supports diverse genres and tasks
Abstract
We wish to measure the information coverage of an ad hoc retrieval algorithm, that is, how much of the range of available relevant information is covered by the search results. Information coverage is a central aspect for retrieval, especially when the retrieval system is integrated with generative models in a retrieval-augmented generation (RAG) system. The classic metrics for ad hoc retrieval, precision and recall, reward a system as more and more relevant documents are retrieved. However, since relevance in ad hoc test collections is defined for a document without any relation to other documents that might contain the same information, high recall is sufficient but not necessary to ensure coverage. The same is true for other metrics such as rank-biased precision (RBP), normalized discounted cumulative gain (nDCG), and mean average precision (MAP). Test collections developed around…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsInformation Retrieval and Search Behavior · Image Retrieval and Classification Techniques · Face recognition and analysis
