CUB: Benchmarking Context Utilisation Techniques for Language Models

TL;DR

This paper introduces CUB, a comprehensive benchmark for evaluating context utilisation techniques in language models, revealing their limitations across diverse noisy contexts and datasets.

Contribution

The paper presents the first systematic benchmark for CMTs, providing extensive evaluation and exposing gaps in current methods' robustness and generalization.

Findings

Most CMTs struggle with diverse real-world contexts.

Many methods perform well on synthetic but not real datasets.

Current evaluation practices are insufficient for comprehensive assessment.

Abstract

Incorporating external knowledge is crucial for knowledge-intensive tasks, such as question answering and fact checking. However, language models (LMs) may ignore relevant information that contradicts outdated parametric memory or be distracted by irrelevant contexts. While many context utilisation manipulation techniques (CMTs) have recently been proposed to alleviate these issues, few have seen systematic comparison. In this paper, we develop CUB (Context Utilisation Benchmark) - the first comprehensive benchmark designed to help diagnose CMTs under diverse noisy context conditions within retrieval-augmented generation (RAG). With this benchmark, we conduct the most extensive evaluation to date of seven state-of-the-art methods, representative of the main categories of CMTs, across three diverse datasets and tasks, applied to 11 LMs. Our findings expose critical gaps in current CMT evaluation practices, demonstrating the need for holistic testing. We reveal that most existing CMTs struggle to handle the full spectrum of context types encountered in real-world RAG scenarios. We also find that many CMTs display inflated performance on simple synthesised datasets, compared to more realistic datasets with naturally occurring samples.