Oolong: Evaluating Long Context Reasoning and Aggregation Capabilities

TL;DR

Oolong is a benchmark designed to evaluate long-context reasoning and aggregation capabilities in models, highlighting current limitations even in state-of-the-art models with extensive context lengths.

Contribution

The paper introduces Oolong, a new benchmark for long-context reasoning tasks that require analyzing and aggregating information from large text chunks, including synthetic and real-world data.

Findings

Frontier models perform poorly on Oolong, with less than 50% accuracy at 128K context length.

Oolong challenges models to perform classification, counting, and reasoning over temporal and user relations.

The authors release data and tools to foster development of better long-context reasoning models.

Abstract

As model context lengths continue to grow, concerns about whether models effectively use the full context length have persisted. While several carefully designed long-context evaluations have recently been released, these evaluations tend to rely on retrieval from one or more sections of the context, which allows nearly all of the context tokens to be disregarded as noise. This represents only one type of task that might be performed with long context. We introduce Oolong, a benchmark of long-context reasoning tasks that require analyzing individual chunks of text on an atomic level, and then aggregating these analyses to answer distributional questions. Oolong is separated into two task sets: Oolong-synth, a set of naturalistic synthetic tasks, where we can easily ablate components of the reasoning problem; and Oolong-real, a downstream setting which requires reasoning over real-world conversational data. Oolong requires models to reason over large quantities of examples, to perform both classification and counting in-context, and to reason over temporal and user relations. Even frontier models struggle on Oolong, with GPT-5, Claude-Sonnet-4, and Gemini-2.5-Pro all achieving less than 50% accuracy on both splits at 128K. We release the data and evaluation harness for Oolong to enable further development of models that can reason over large quantities of text.