Think Longer to Explore Deeper: Learn to Explore In-Context via Length-Incentivized Reinforcement Learning

TL;DR

This paper introduces Length-Incentivized Exploration, a reinforcement learning approach that encourages models to generate longer reasoning trajectories, thereby enhancing their in-context exploration capabilities and improving performance on various tasks.

Contribution

The paper proposes a novel length-based reward and redundancy penalty to overcome the shallow exploration trap, significantly improving in-context exploration in language models.

Findings

Achieves 4.4% improvement on in-domain tasks

Achieves 2.7% improvement on out-of-domain benchmarks

Effectively incentivizes longer, more diverse reasoning trajectories

Abstract

Achieving effective test-time scaling requires models to engage in In-Context Exploration -- the intrinsic ability to generate, verify, and refine multiple reasoning hypotheses within a single continuous context. Grounded in State Coverage theory, our analysis identifies a critical bottleneck to enabling this capability: while broader state coverage requires longer reasoning trajectories, the probability of sampling such sequences decays exponentially during autoregressive generation, a phenomenon we term the ``Shallow Exploration Trap''. To bridge this gap, we propose Length-Incentivized Exploration(\method). This simple yet effective recipe explicitly encourages models to explore more via a length-based reward coupled with a redundancy penalty, thereby maximizing state coverage in two-step manner. Comprehensive experiments across different models (Qwen3, Llama) demonstrate that \method effectively incentivize in-context exploration. As a result, our method achieves an average improvement of 4.4\% on in-domain tasks and a 2.7\% gain on out-of-domain benchmarks.