A Reality Check on Context Utilisation for Retrieval-Augmented Generation

TL;DR

This paper introduces DRUID, a new dataset with real-world contexts for claim verification, revealing that synthetic datasets often misrepresent real retrieval complexity and inflate context utilisation results.

Contribution

The paper presents DRUID, a real-world dataset for evaluating context utilisation in RAG, and demonstrates the limitations of synthetic datasets in representing real retrieval challenges.

Findings

Synthetic datasets exaggerate rare context features.

Artificial datasets inflate context utilisation scores.

Context source properties correlate more with ACU than singleton characteristics.

Abstract

Retrieval-augmented generation (RAG) helps address the limitations of parametric knowledge embedded within a language model (LM). In real world settings, retrieved information can vary in complexity, yet most investigations of LM utilisation of context has been limited to synthetic text. We introduce DRUID (Dataset of Retrieved Unreliable, Insufficient and Difficult-to-understand contexts) with real-world queries and contexts manually annotated for stance. The dataset is based on the prototypical task of automated claim verification, for which automated retrieval of real-world evidence is crucial. We compare DRUID to synthetic datasets (CounterFact, ConflictQA) and find that artificial datasets often fail to represent the complexity and diversity of realistically retrieved context. We show that synthetic datasets exaggerate context characteristics rare in real retrieved data, which leads to inflated context utilisation results, as measured by our novel ACU score. Moreover, while previous work has mainly focused on singleton context characteristics to explain context utilisation, correlations between singleton context properties and ACU on DRUID are surprisingly small compared to other properties related to context source. Overall, our work underscores the need for real-world aligned context utilisation studies to represent and improve performance in real-world RAG settings.