MAD-OPD: Breaking the Ceiling in On-Policy Distillation via Multi-Agent Debate

TL;DR

MAD-OPD introduces a multi-agent debate framework to enhance on-policy distillation, surpassing single-teacher limits and stabilizing training in agentic tasks, achieving state-of-the-art results across multiple benchmarks.

Contribution

It proposes MAD-OPD, a novel multi-agent debate approach for on-policy distillation, and extends it to agentic tasks with a new divergence principle, improving performance significantly.

Findings

MAD-OPD outperforms all six baseline configurations.

It improves agentic average by 2.4% and code average by 3.7%.

The method ranks first across all tested configurations.

Abstract

On-policy distillation (OPD) trains a student on its own trajectories under token-level teacher supervision, but existing methods are capped by a single-teacher capability ceiling: when the teacher errs, the student inherits the error. OPD also remains largely unexplored in agentic tasks, where per-step errors compound across long trajectories and destabilize training. We propose MAD-OPD (Multi-Agent Debate-driven On-Policy Distillation), which breaks this ceiling by recasting the distillation teacher as a deliberative collective of teachers that debate over the student's on-policy state; the debate produces an emergent collective intelligence that supplies token-level supervision, with each teacher's contribution weighted by its post-debate confidence. To extend OPD to agentic tasks, we also introduce On-Policy Agentic Distillation (OPAD), which adds step-level sampling to stabilize training under multi-step error compounding. We additionally derive a task-adaptive divergence principle, selecting JSD (Jensen-Shannon divergence) for agentic stability and reverse KL (Kullback-Leibler) divergence for code generation, and verify it both theoretically and empirically. Across six teacher-student configurations (Qwen3 and Qwen3.5; 1.7B-14B students, 8B-32B teachers) and five agentic and code benchmarks, MAD-OPD ranks first across all six configurations; on the 14B+8B$\to$4B setting it lifts the agentic average by $+2.4\%$ and the code average by $+3.7\%$ over the stronger single-teacher OPD.