Open-World Evaluation for Retrieving Diverse Perspectives

TL;DR

This paper introduces BERDS, a benchmark for evaluating retrieval systems on their ability to surface diverse perspectives on subjective questions, using a language model-based evaluator to assess perspective coverage.

Contribution

The paper presents a new benchmark and evaluation methodology for retrieving diverse perspectives, addressing the challenge of subjective relevance beyond string matching.

Findings

Existing retrievers cover only 40% of perspectives.

Query expansion and reranking improve diversity.

Language model evaluator effectively assesses perspective relevance.

Abstract

We study retrieving a set of documents that covers various perspectives on a complex and contentious question (e.g., will ChatGPT do more harm than good?). We curate a Benchmark for Retrieval Diversity for Subjective questions (BERDS), where each example consists of a question and diverse perspectives associated with the question, sourced from survey questions and debate websites. On this data, retrievers paired with a corpus are evaluated to surface a document set that contains diverse perspectives. Our framing diverges from most retrieval tasks in that document relevancy cannot be decided by simple string matches to references. Instead, we build a language model-based automatic evaluator that decides whether each retrieved document contains a perspective. This allows us to evaluate the performance of three different types of corpus (Wikipedia, web snapshot, and corpus constructed on the fly with retrieved pages from the search engine) paired with retrievers. Retrieving diverse documents remains challenging, with the outputs from existing retrievers covering all perspectives on only 40% of the examples. We further study the effectiveness of query expansion and diversity-focused reranking approaches and analyze retriever sycophancy.