LongR: Unleashing Long-Context Reasoning via Reinforcement Learning with Dense Utility Rewards

TL;DR

LongR introduces a reinforcement learning framework that improves long-context reasoning in language models by using dense utility rewards and a dynamic reasoning mechanism, leading to significant performance gains.

Contribution

The paper presents LongR, a novel RL-based approach that integrates a reasoning mechanism with dense utility rewards to enhance long-context reasoning capabilities.

Findings

Achieves 9% improvement on LongBench v2

Consistently improves performance across multiple RL algorithms

Enhances robustness against distractors in reasoning tasks

Abstract

Reinforcement Learning has emerged as a key driver for LLM reasoning. This capability is equally pivotal in long-context scenarios--such as long-dialogue understanding and structured data analysis, where the challenge extends beyond consuming tokens to performing rigorous deduction. While existing efforts focus on data synthesis or architectural changes, recent work points out that relying solely on sparse, outcome-only rewards yields limited gains, as such coarse signals are often insufficient to effectively guide the complex long-context reasoning. To address this, we propose LongR, a unified framework that enhances long-context performance by integrating a dynamic "Think-and-Read" mechanism, which interleaves reasoning with document consultation, with a contextual density reward based on relative information gain to quantify the utility of the relevant documents. Empirically, LongR achieves a 9% gain on LongBench v2 and consistent improvements on RULER and InfiniteBench, demonstrating robust efficiency in navigating extensive contexts. Furthermore, LongR consistently enhances performance across diverse RL algorithms (e.g., DAPO, GSPO). Finally, we conduct in-depth analyses to investigate the impact of reasoning chain length on efficiency and the model's robustness against distractors.