Evaluating Language Model Context Windows: A "Working Memory" Test and Inference-time Correction

TL;DR

This paper introduces SWiM, a new benchmark for evaluating long context language models, revealing performance degradation in middle-of-context information, and proposes medoid voting as an effective correction method.

Contribution

The paper presents SWiM, a novel evaluation framework for long context models, and introduces medoid voting, a simple inference-time correction technique to improve accuracy.

Findings

Performance drops when information is in the middle of the context.

Medoid voting improves accuracy by up to 24%.

Even strong models like GPT-4 are affected by the lost-in-the-middle effect.

Abstract

Large language models are prominently used in real-world applications, often tasked with reasoning over large volumes of documents. An exciting development in this space is models boasting extended context capabilities, with some accommodating over 2 million tokens. Such long context model capabilities remain uncertain in production systems, motivating the need to benchmark their performance on real world use cases. We address this challenge by proposing SWiM, an evaluation framework that addresses the limitations of standard tests. Testing the framework on eight long context models, we find that even strong models such as GPT-4 and Claude 3 Opus degrade in performance when information is present in the middle of the context window (lost-in-the-middle effect). Next, in addition to our benchmark, we propose medoid voting, a simple, but effective training-free approach that helps alleviate this effect, by generating responses a few times, each time randomly permuting documents in the context, and selecting the medoid answer. We evaluate medoid voting on single document QA tasks, achieving up to a 24% lift in accuracy. Our code is available at https://github.com/snorkel-ai/long-context-eval.