OPSD Compresses What RLVR Teaches: A Post-RL Compaction Stage for Reasoning Models

TL;DR

This paper investigates OPSD's role in compressing reasoning models, finding it acts more reliably as a compression tool rather than a correction method in mathematical reasoning tasks.

Contribution

It reveals OPSD's effectiveness as a compression mechanism in reasoning models and proposes a revised post-training pipeline for improved performance.

Findings

OPSD preserves accuracy when trained on correct rollouts while shortening responses.

Training OPSD on incorrect rollouts damages model accuracy.

A new pipeline combining SFT, RLVR, and OPSD improves reasoning model training.

Abstract

On-Policy Self-Distillation (OPSD) has recently emerged as an alternative to Reinforcement Learning with Verifiable Rewards (RLVR), promising higher accuracy and shorter responses through token-level credit assignment from a self-teacher conditioned on privileged context. However, this promise does not carry over to thinking-enabled mathematical reasoning, where reported accuracy gains shrink and sometimes turn negative. We hypothesize that hindsight supervision can specify better token-level alternatives in short thinking-disabled outputs, but in long thinking-enabled traces it more readily identifies redundancy than supplies better replacements. To test this, we applied OPSD separately to correct and incorrect rollout groups, so that compression and correction can be observed in isolation. Our results show that in thinking-enabled mathematical reasoning, OPSD behaves most reliably as a compression mechanism rather than a correction mechanism: training only on correct rollouts preserves accuracy while substantially shortening responses, whereas training only on incorrect rollouts damages accuracy. In light of these findings, we propose a revised post-training pipeline for thinking-enabled mathematical reasoning: SFT then RLVR then OPSD.