Contextual Relevance and Adaptive Sampling for LLM-Based Document Reranking

TL;DR

This paper introduces a new approach for document reranking that accounts for context-dependent relevance, using a sampling-based algorithm called TS-SetRank, which improves retrieval quality for complex queries.

Contribution

It proposes the concept of contextual relevance and develops TS-SetRank, an uncertainty-aware sampling method that enhances LLM-based reranking performance.

Findings

TS-SetRank improves nDCG@10 by 15-25% on BRIGHT.

Modeling relevance as context-dependent boosts reranking accuracy.

Batch composition influences reranking performance significantly.

Abstract

Reranking algorithms have made progress in improving document retrieval quality by efficiently aggregating relevance judgments generated by large language models (LLMs). However, identifying relevant documents for queries that require in-depth reasoning remains a major challenge. Reasoning-intensive queries often exhibit multifaceted information needs and nuanced interpretations, rendering document relevance inherently context dependent. To address this, we propose contextual relevance, which we define as the probability that a document is relevant to a given query, marginalized over the distribution of different reranking contexts it may appear in (i.e., the set of candidate documents it is ranked alongside and the order in which the documents are presented to a reranking model). While prior works have studied methods to mitigate the positional bias LLMs exhibit by accounting for the ordering of documents, we empirically find that the compositions of these batches also plays an important role in reranking performance. To efficiently estimate contextual relevance, we propose TS-SetRank, a sampling-based, uncertainty-aware reranking algorithm. Empirically, TS-SetRank improves nDCG@10 over retrieval and reranking baselines by 15-25% on BRIGHT and 6-21% on BEIR, highlighting the importance of modeling relevance as context-dependent.