FRESCO: Benchmarking and Optimizing Re-rankers for Evolving Semantic Conflict in Retrieval-Augmented Generation

TL;DR

FRESCO introduces a benchmark for evaluating re-rankers in dynamic, evolving information contexts, revealing biases and proposing instruction-based improvements to better prioritize recent evidence in retrieval-augmented generation.

Contribution

The paper presents FRESCO, a novel benchmark for assessing re-rankers in temporally evolving scenarios, and proposes an instruction optimization framework to improve their recency bias.

Findings

Existing re-rankers favor older, semantically rich documents over recent, factual evidence.

FRESCO effectively tests re-rankers' ability to prioritize recent information in dynamic settings.

Instruction optimization yields up to 27% improvement on Evolving Knowledge tasks.

Abstract

Retrieval-Augmented Generation (RAG) is a key approach to mitigating the temporal staleness of large language models (LLMs) by grounding responses in up-to-date evidence. Within the RAG pipeline, re-rankers play a pivotal role in selecting the most useful documents from retrieved candidates. However, existing benchmarks predominantly evaluate re-rankers in static settings and do not adequately assess performance under evolving information -- a critical gap, as real-world systems often must choose among temporally different pieces of evidence. To address this limitation, we introduce FRESCO (Factual Recency and Evolving Semantic COnflict), a benchmark for evaluating re-rankers in temporally dynamic contexts. By pairing recency-seeking queries with historical Wikipedia revisions, FRESCO tests whether re-rankers can prioritize factually recent evidence while maintaining semantic relevance. Our evaluation reveals a consistent failure mode across existing re-rankers: a strong bias toward older, semantically rich documents, even when they are factually obsolete. We further investigate an instruction optimization framework to mitigate this issue. By identifying Pareto-optimal instructions that balance Evolving and Non-Evolving Knowledge tasks, we obtain gains of up to 27% on Evolving Knowledge tasks while maintaining competitive performance on Non-Evolving Knowledge tasks.