Multi-Rollout On-Policy Distillation via Peer Successes and Failures

TL;DR

This paper introduces MOPD, a peer-conditioned on-policy distillation method that leverages multiple student rollouts, including successes and failures, to provide more informative training signals for large language models.

Contribution

MOPD is a novel framework that uses peer rollouts to improve on-policy distillation by incorporating both positive and negative evidence, enhancing model training.

Findings

MOPD outperforms standard on-policy baselines across multiple benchmarks.

Mixed success-failure contexts improve alignment with verifier rewards.

Exploiting multi-rollout behavior leads to more faithful supervision.

Abstract

Large language models are often post-trained with sparse verifier rewards, which indicate whether a sampled trajectory succeeds but provide limited guidance about where reasoning succeeds or fails. On-policy distillation (OPD) offers denser token-level supervision by training on student-generated trajectories, yet existing methods typically distill each rollout independently and ignore the other attempts sampled for the same prompt. We introduce Multi-Rollout On-Policy Distillation (MOPD), a peer-conditioned distillation framework that uses the student's local rollout group to construct more informative teacher signals. MOPD conditions the teacher on both successful and failed peer rollouts: successes provide positive evidence for valid reasoning patterns, while failures provide structured negative evidence about plausible mistakes to avoid. We study two peer-context constructions: positive peer imitation and contrastive success-failure conditioning. Experiments on competitive programming, mathematical reasoning, scientific question answering, and tool-use benchmarks show that MOPD consistently improves over standard on-policy baselines. Further teacher-signal analysis shows that mixed success-failure contexts better align teacher scores with verifier rewards, indicating that the gains arise from more faithful, instance-adaptive supervision. These results indicate that effective on-policy distillation should exploit the student's multi-rollout trial-and-error behavior rather than treating rollouts as isolated samples.