OVD: On-policy Verbal Distillation

TL;DR

OVD introduces a memory-efficient on-policy verbal distillation framework that replaces token-level matching with trajectory matching using verbal scores, enabling better exploration and improved performance in reasoning tasks.

Contribution

The paper proposes a novel on-policy verbal distillation method that reduces memory usage and enhances exploration by replacing token-level alignment with trajectory-level matching using verbal scores.

Findings

Significant performance improvements on Web Q&A and math reasoning tasks.

Reduced memory consumption compared to existing token-level distillation methods.

Faster training efficiency with comparable or better results.

Abstract

Knowledge distillation offers a promising path to transfer reasoning capabilities from large teacher models to efficient student models; however, existing token-level on-policy distillation methods require token-level alignment between the student and teacher models, which restricts the student model's exploration ability, prevent effective use of interactive environment feedback, and suffer from severe memory bottlenecks in reinforcement learning. We introduce On-policy Verbal Distillation (OVD), a memory-efficient framework that replaces token-level probability matching with trajectory matching using discrete verbal scores (0--9) from teacher models. OVD dramatically reduces memory consumption while enabling on-policy distillation from teacher models with verbal feedback, and avoids token-level alignment, allowing the student model to freely explore the output space. Extensive experiments on Web question answering and mathematical reasoning tasks show that OVD substantially outperforms existing methods, delivering up to +12.9% absolute improvement in average EM on Web Q&A tasks and a up to +25.7% gain on math benchmarks (when trained with only one random samples), while also exhibiting superior training efficiency. Our project page is available at https://OVD.github.io