Fast and Effective On-policy Distillation from Reasoning Prefixes

TL;DR

This paper introduces an on-policy prefix distillation method that reduces training costs by focusing on prefixes of student outputs, maintaining performance while significantly decreasing FLOP consumption.

Contribution

It proposes a simple modification to on-policy distillation that applies the distillation objective only to output prefixes, greatly reducing training costs without sacrificing accuracy.

Findings

Achieves comparable performance to full OPD

Reduces training FLOP by up to 47 times

Effective on AI-for-Math and out-of-domain benchmarks

Abstract

On-policy distillation (OPD), which samples trajectories from the student model and supervises them with a teacher at the token level, avoids relying solely on verifiable terminal rewards and can yield better generalization than off-policy distillation. However, OPD requires expensive on-the-fly sampling of the student policy during training, which substantially increases training cost, especially for long responses. Our initial analysis shows that, during OPD, training signals are often concentrated in the prefix of each output, and that even a short teacher-generated prefix can significantly help the student produce the correct answer. Motivated by these observations, we propose a simple yet effective modification of OPD: we apply the distillation objective only to prefixes of student-generated outputs and terminate each sampling early during distillation. Experiments on a suite of AI-for-Math and out-of-domain benchmarks show that on-policy prefix distillation matches the performance of full OPD while reducing training FLOP by 2x-47x.