Dissecting Long-Chain-of-Thought Reasoning Models: An Empirical Study

TL;DR

This paper empirically analyzes the training dynamics of long-chain-of-thought reasoning models, revealing insights into sample roles, data inefficiencies, and factors affecting model stability and performance.

Contribution

It provides a systematic analysis of positive and negative samples in RL training, proposes strategies to improve data efficiency, and investigates causes of performance instability in reasoning models.

Findings

Negative samples enhance generalization and robustness.

Training on negative samples alone can achieve strong reasoning performance.

Strategies like relative length rewards improve data efficiency.

Abstract

Despite recent progress in training long-chain-of-thought reasoning models via scaling reinforcement learning (RL), its underlying training dynamics remain poorly understood, and several counterintuitive behaviors persist. This work focuses on three key aspects: (1) We systematically analyze the roles of positive and negative samples in scaling RL, revealing that positive samples mainly facilitate precise fitting to the training data, whereas negative samples significantly enhance generalization and robustness. Interestingly, while positive samples are essential for convergence in the zero-RL setting, training on negative samples alone suffices to attain strong reasoning performance and even better generalization in cold-start scenarios. (2) We identify substantial data inefficiency in group relative policy optimization, where over half of the samples yield zero advantage. To address this, we explore two strategies, including relative length rewards and offline sample injection, to leverage these data better and enhance reasoning efficiency and capability. (3) We investigate unstable performance across various reasoning models and benchmarks, attributing instability to uncertain problems with ambiguous outcomes, and demonstrate that greedy decoding can distort evaluation by flipping the correctness of responses. Our code is available at: https://github.com/takagi97/Dissect-Long-Reason-Models.