SODA: Semi On-Policy Black-Box Distillation for Large Language Models

TL;DR

SODA introduces a semi on-policy distillation method for large language models that achieves high-quality results efficiently by pairing teacher responses with static student outputs, avoiding costly dynamic rollouts.

Contribution

The paper presents SODA, a novel distillation approach that eliminates adversarial training and reduces computational costs while maintaining or improving performance.

Findings

SODA matches or outperforms state-of-the-art methods on 15 out of 16 benchmarks.

It trains 10 times faster and uses 27% less GPU memory than previous methods.

It completely eliminates adversarial instability in distillation.

Abstract

Black-box knowledge distillation for large language models presents a strict trade-off. Simple off-policy methods (e.g., sequence-level knowledge distillation) struggle to correct the student's inherent errors. Fully on-policy methods (e.g., Generative Adversarial Distillation) solve this via adversarial training but introduce well-known training instability and crippling computational overhead. To address this dilemma, we propose SODA (Semi On-policy Distillation with Alignment), a highly efficient alternative motivated by the inherent capability gap between frontier teachers and much smaller base models. Because a compact student model's natural, zero-shot responses are almost strictly inferior to the powerful teacher's targets, we can construct a highly effective contrastive signal simply by pairing the teacher's optimal response with a one-time static snapshot of the student's outputs. This demonstrates that exposing the small student to its own static inferior behaviors is sufficient for high-quality distribution alignment, eliminating the need for costly dynamic rollouts and fragile adversarial balancing. Extensive evaluations across four compact Qwen2.5 and Llama-3 models validate this semi on-policy paradigm. SODA matches or outperforms the state-of-the-art methods on 15 out of 16 benchmark results. More importantly, it achieves this superior distillation quality while training 10 times faster, consuming 27% less peak GPU memory, and completely eliminating adversarial instability.