Longer Context, Deeper Thinking: Uncovering the Role of Long-Context Ability in Reasoning

TL;DR

This paper investigates how enhancing long-context capacity in language models improves reasoning abilities, demonstrating that stronger long-context modeling leads to better performance across reasoning benchmarks, even with short inputs.

Contribution

It provides empirical evidence that increasing long-context capacity before fine-tuning enhances reasoning performance, highlighting long-context ability as a key factor in model design.

Findings

Models with stronger long-context capacity outperform others in reasoning benchmarks.

Long-context training benefits general reasoning performance, even on short input tasks.

Enhancing long-context ability is crucial for developing more capable reasoning models.

Abstract

Recent language models exhibit strong reasoning capabilities, yet the influence of long-context capacity on reasoning remains underexplored. In this work, we hypothesize that current limitations in reasoning stem, in part, from insufficient long-context capacity, motivated by empirical observations such as (1) higher context window length often leads to stronger reasoning performance, and (2) failed reasoning cases resemble failed long-context cases. To test this hypothesis, we examine whether enhancing a model's long-context ability before Supervised Fine-Tuning (SFT) leads to improved reasoning performance. Specifically, we compared models with identical architectures and fine-tuning data but varying levels of long-context capacity. Our results reveal a consistent trend: models with stronger long-context capacity achieve significantly higher accuracy on reasoning benchmarks after SFT. Notably, these gains persist even on tasks with short input lengths, indicating that long-context training offers generalizable benefits for reasoning performance. These findings suggest that long-context modeling is not just essential for processing lengthy inputs, but also serves as a critical foundation for reasoning. We advocate for treating long-context capacity as a first-class objective in the design of future language models.