BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval

TL;DR

BRIGHT is a new benchmark designed to evaluate retrieval models on complex, reasoning-intensive queries across diverse domains, revealing significant performance gaps and highlighting the importance of explicit reasoning and retrieval-augmented methods.

Contribution

This paper introduces BRIGHT, the first benchmark focusing on reasoning-intensive retrieval tasks with real-world queries, and demonstrates the limitations of current models while proposing reasoning-enhanced retrieval strategies.

Findings

State-of-the-art models perform poorly on BRIGHT

Explicit reasoning improves retrieval accuracy by up to 12.2 points

Retrieval-augmented methods enhance question-answering performance

Abstract

Existing retrieval benchmarks primarily consist of information-seeking queries (e.g., aggregated questions from search engines) where keyword or semantic-based retrieval is usually sufficient. However, many complex real-world queries require in-depth reasoning to identify relevant documents that go beyond surface form matching. For example, finding documentation for a coding question requires understanding the logic and syntax of the functions involved. To better benchmark retrieval on such challenging queries, we introduce BRIGHT, the first text retrieval benchmark that requires intensive reasoning to retrieve relevant documents. Our dataset consists of 1,384 real-world queries spanning diverse domains, such as economics, psychology, mathematics, and coding. These queries are drawn from naturally occurring and carefully curated human data. Extensive evaluation reveals that even state-of-the-art retrieval models perform poorly on BRIGHT. The leading model on the MTEB leaderboard (Muennighoff et al., 2023) SFR-Embedding-Mistral (Meng et al., 2024), which achieves a score of 59.0 nDCG@10,1 produces a score of nDCG@10 of 18.3 on BRIGHT. We show that incorporating explicit reasoning about the query improves retrieval performance by up to 12.2 points. Moreover, incorporating retrieved documents from the top-performing retriever boosts question-answering performance. We believe that BRIGHT paves the way for future research on retrieval systems in more realistic and challenging settings.