Entropy-Aware On-Policy Distillation of Language Models

TL;DR

This paper introduces an entropy-aware on-policy distillation method for language models that balances mode-seeking and mode-covering objectives, leading to improved diversity and accuracy in student models.

Contribution

It proposes augmenting reverse KL with forward KL based on teacher entropy, enhancing knowledge transfer by maintaining diversity and stability.

Findings

Maintains generation diversity with high token entropy.

Improves student-teacher alignment on high-entropy tokens.

Achieves significant accuracy gains across multiple benchmarks.

Abstract

On-policy distillation is a promising approach for transferring knowledge between language models, where a student learns from dense token-level signals along its own trajectories. This framework typically uses reverse KL divergence, encouraging the student to match the teacher's high-confidence predictions. However, we show that the mode-seeking property of reverse KL reduces generation diversity and yields unstable learning signals when the teacher distribution has high entropy. To address this, we introduce Entropy-Aware On-Policy Distillation. Our key idea is augmenting the standard reverse KL objective with forward KL when teacher entropy is high, capturing the full range of plausible outputs while retaining precise imitation elsewhere. It balances mode-seeking precision with mode-covering robustness without sacrificing on-policy training efficiency. Experiments show that our method maintains generation diversity (sustained token-level entropy) and improves student-teacher alignment (lower forward KL on high-entropy tokens). Across six math reasoning benchmarks, this yields Pass@8 accuracy gains of +1.37 for Qwen3-0.6B-Base, +2.39 for Qwen3-1.7B-Base, and +5.05 for Qwen3-4B-Base compared to baseline on-policy distillation methods. These results demonstrate that accounting for teacher uncertainty is essential for maintaining diversity and achieving effective knowledge transfer.