Learning Structured Reasoning via Tractable Trajectory Control

TL;DR

This paper introduces Ctrl-R, a framework that guides large language models to explore and learn diverse structured reasoning patterns, improving their problem-solving abilities in complex reasoning tasks.

Contribution

The paper proposes a novel reinforcement learning approach with tractable trajectory control to systematically discover and reinforce diverse reasoning behaviors in language and vision-language models.

Findings

Enhanced exploration of reasoning patterns in models.

Improved performance on mathematical reasoning tasks.

Stable optimization with importance-sampling techniques.

Abstract

Large language models can exhibit emergent reasoning behaviors, often manifested as recurring lexical patterns (e.g., "wait," indicating verification). However, complex reasoning trajectories remain sparse in unconstrained sampling, and standard RL often fails to guarantee the acquisition of diverse reasoning behaviors. We propose a systematic discovery and reinforcement of diverse reasoning patterns through structured reasoning, a paradigm that requires targeted exploration of specific reasoning patterns during the RL process. To this end, we propose Ctrl-R, a framework for learning structured reasoning via tractable trajectory control that actively guides the rollout process, incentivizing the exploration of diverse reasoning patterns that are critical for complex problem-solving. The resulting behavior policy enables accurate importance-sampling estimation, supporting unbiased on-policy optimization. We further introduce a power-scaling factor on the importance-sampling weights, allowing the policy to selectively learn from exploratory, out-of-distribution trajectories while maintaining stable optimization. Experiments demonstrate that Ctrl-R enables effective exploration and internalization of previously unattainable reasoning patterns, yielding consistent improvements across language and vision-language models on mathematical reasoning tasks.