Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe

TL;DR

Uni-OPD introduces a unified framework for on-policy distillation that enhances exploration and supervision reliability, improving model performance across diverse large language and multimodal models.

Contribution

The paper proposes Uni-OPD, a dual-perspective optimization strategy that addresses key bottlenecks in on-policy distillation for LLMs and MLLMs, demonstrating broad applicability.

Findings

Effective in 5 domains and 16 benchmarks.

Improves exploration of informative states during training.

Restores order consistency in teacher supervision.

Abstract

On-policy distillation (OPD) has recently emerged as an effective post-training paradigm for consolidating the capabilities of specialized expert models into a single student model. Despite its empirical success, the conditions under which OPD yields reliable improvement remain poorly understood. In this work, we identify two fundamental bottlenecks that limit effective OPD: insufficient exploration of informative states and unreliable teacher supervision for student rollouts. Building on this insight, we propose Uni-OPD, a unified OPD framework that generalizes across Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs), centered on a dual-perspective optimization strategy. Specifically, from the student's perspective, we adopt two data balancing strategies to promote exploration of informative student-generated states during training. From the teacher's perspective, we show that reliable supervision hinges on whether aggregated token-level guidance remains order-consistent with the outcome reward. To this end, we develop an outcome-guided margin calibration mechanism to restore order consistency between correct and incorrect trajectories. We conduct extensive experiments on 5 domains and 16 benchmarks covering diverse settings, including single-teacher and multi-teacher distillation across LLMs and MLLMs, strong-to-weak distillation, and cross-modal distillation. Our results verify the effectiveness and versatility of Uni-OPD and provide practical insights into reliable OPD.