Near-Policy: Accelerating On-Policy Distillation via Asynchronous Generation and Selective Packing

TL;DR

Near-Policy Distillation (NPD) accelerates on-policy model training by decoupling generation and training, using selective filtering and sparse updates to maintain stability and improve efficiency, achieving significant speedups and performance gains.

Contribution

The paper introduces NPD, an asynchronous distillation framework with novel filtering and update strategies that enhance efficiency and stability in on-policy autoregressive model training.

Findings

Achieves 8.1x speedup over on-policy baselines.

Outperforms supervised fine-tuning by 8.09%.

Enables openPangu-Embedded-1B to reach a state-of-the-art score of 68.73%.

Abstract

Standard knowledge distillation for autoregressive models often suffers from distribution mismatch. While on-policy methods mitigate this by leveraging student-generated outputs, they rely on computationally expensive Reinforcement Learning (RL) frameworks. To improve efficiency, we propose Near-Policy Distillation (NPD), an asynchronous approach that decouples student generation from training. This reformulation enables Supervised Fine-Tuning (SFT) with sequence packing. However, asynchronous updates inevitably introduce policy lag and sample noise, which can cause the behavior to drift from near-policy toward off-policy. To counteract this without sacrificing efficiency, NPD integrates sparse student updates and the $\Delta$-IFD filtering mechanism, a heuristic sample selection mechanism that empirically stabilizes the optimization trajectory. By filtering extreme out-of-distribution samples, $\Delta$-IFD prevents noise from dominating the gradients, ensuring updates remain within a safe proximal learning zone. Empirically, the NPD framework achieves a 8.1x speedup over on-policy baselines and outperforms SFT by 8.09%. Crucially, by effectively narrowing the exploration space for subsequent RL, our method enables openPangu-Embedded-1B to reach a state-of-the-art score of 68.73%, outperforming the substantially larger Qwen3-1.7B. Codes will be released soon.