A Controllable Examination for Long-Context Language Models

TL;DR

This paper introduces LongBioBench, a controllable and interpretable benchmark using artificially generated biographies to evaluate long-context language models across understanding, reasoning, and trustworthiness, addressing limitations of existing benchmarks.

Contribution

The study presents LongBioBench, a novel benchmark that offers a controllable, sound, and interpretable environment for evaluating long-context language models, improving upon existing synthetic benchmarks.

Findings

Most models show deficiencies in semantic understanding and reasoning.

Model trustworthiness decreases as context length increases.

Existing benchmarks are vulnerable due to non-coherence and lack of distractors.

Abstract

Existing frameworks for evaluating long-context language models (LCLM) can be broadly categorized into real-world applications (e.g, document summarization) and synthetic tasks (e.g, needle-in-a-haystack). Despite their utility, both approaches are accompanied by certain intrinsic limitations. Real-world tasks often involve complexity that makes interpretation challenging and suffer from data contamination, whereas synthetic tasks frequently lack meaningful coherence between the target information (needle) and its surrounding context (haystack), undermining their validity as proxies for realistic applications. In response to these challenges, we posit that an ideal long-context evaluation framework should be characterized by three essential features: 1) seamless context 2) controllable setting and 3) sound evaluation. This study introduces $\textbf{LongBioBench}$, a benchmark that utilizes artificially generated biographies as a controlled environment for assessing LCLMs across dimensions of understanding, reasoning, and trustworthiness. Our experimental evaluation, which includes 18 LCLMs in total, demonstrates that most models still exhibit deficiencies in semantic understanding and elementary reasoning over retrieved results and are less trustworthy as context length increases. Our further analysis indicates some design choices employed by existing synthetic benchmarks, such as contextual non-coherence, numerical needles, and the absence of distractors, rendering them vulnerable to test the model's long-context capabilities. To sum up, compared to previous synthetic benchmarks, LongBioBench achieves a better trade-off between mirroring authentic language tasks and maintaining controllability, and is highly interpretable and configurable.