SSPO: Self-traced Step-wise Preference Optimization for Process Supervision and Reasoning Compression

TL;DR

SSPO is a novel reinforcement learning framework that improves large language model reasoning by optimizing each step using self-generated preferences, reducing overthinking and enhancing reasoning accuracy and conciseness.

Contribution

It introduces a pluggable, self-supervised RL method that optimizes reasoning steps without auxiliary models or manual annotations, improving efficiency and reasoning quality.

Findings

Reasoning sequences are more accurate and succinct with SSPO.

SSPO effectively reduces overthinking behaviors across multiple domains.

Model performance remains uncompromised while reasoning is compressed.

Abstract

Test-time scaling has proven effective in further enhancing the performance of pretrained Large Language Models (LLMs). However, mainstream post-training methods (i.e., reinforcement learning (RL) with chain-of-thought (CoT) reasoning) often incur substantial computational overhead due to auxiliary models and overthinking. In this paper, we empirically reveal that the incorrect answers partially stem from verbose reasoning processes lacking correct self-fix, where errors accumulate across multiple reasoning steps. To this end, we propose Self-traced Step-wise Preference Optimization (SSPO), a pluggable RL process supervision framework that enables fine-grained optimization of each reasoning step. Specifically, SSPO requires neither auxiliary models nor stepwise manual annotations. Instead, it leverages step-wise preference signals generated by the model itself to guide the optimization process for reasoning compression. Experiments demonstrate that the generated reasoning sequences from SSPO are both accurate and succinct, effectively mitigating overthinking behaviors without compromising model performance across diverse domains and languages.