CorpusQA: A 10 Million Token Benchmark for Corpus-Level Analysis and Reasoning

TL;DR

CorpusQA introduces a large-scale benchmark for evaluating language models' ability to perform reasoning across extensive document collections, highlighting current limitations and proposing new directions.

Contribution

The paper presents a novel 10-million-token benchmark and a data synthesis framework for challenging corpus-level reasoning tasks, improving evaluation and training of long-context models.

Findings

State-of-the-art models struggle with increasing input length.

Standard retrieval methods fail on large, dispersed corpora.

Memory-augmented architectures outperform traditional models.

Abstract

While large language models now handle million-token contexts, their capacity for reasoning across entire document repositories remains largely untested. Existing benchmarks are inadequate, as they are mostly limited to single long texts or rely on a "sparse retrieval" assumption-that answers can be derived from a few relevant chunks. This assumption fails for true corpus-level analysis, where evidence is highly dispersed across hundreds of documents and answers require global integration, comparison, and statistical aggregation. To address this critical gap, we introduce CorpusQA, a new benchmark scaling up to 10 million tokens, generated via a novel data synthesis framework. By decoupling reasoning from textual representation, this framework creates complex, computation-intensive queries with programmatically guaranteed ground-truth answers, challenging systems to perform holistic reasoning over vast, unstructured text without relying on fallible human annotation. We further demonstrate the utility of our framework beyond evaluation, showing that fine-tuning on our synthesized data effectively enhances an LLM's general long-context reasoning capabilities. Extensive experiments reveal that even state-of-the-art long-context LLMs struggle as input length increases, and standard retrieval-augmented generation systems collapse entirely. Our findings indicate that memory-augmented agentic architectures offer a more robust alternative, suggesting a critical shift is needed from simply extending context windows to developing advanced architectures for global information synthesis.