Reinforcement-aware Knowledge Distillation for LLM Reasoning

TL;DR

This paper introduces RLAD, a reinforcement learning-aware distillation method that selectively guides LLMs during RL training to improve reasoning capabilities while balancing exploration and imitation.

Contribution

The paper proposes RLAD with TRRD, a novel distillation approach that aligns teacher-student training during RL, addressing distribution mismatch and objective interference issues.

Findings

RLAD outperforms offline distillation and standard methods on reasoning benchmarks.

TRRD provides advantage-aware, trust-region-bounded distillation.

The approach effectively balances exploration, exploitation, and imitation during training.

Abstract

Reinforcement learning (RL) post-training has recently driven major gains in long chain-of-thought reasoning large language models (LLMs), but the high inference cost of such models motivates distillation into smaller students. Most existing knowledge distillation (KD) methods are designed for supervised fine-tuning (SFT), relying on fixed teacher traces or teacher-student Kullback-Leibler (KL) divergence-based regularization. When combined with RL, these approaches often suffer from distribution mismatch and objective interference: teacher supervision may not align with the student's evolving rollout distribution, and the KL regularizer can compete with reward maximization and require careful loss balancing. To address these issues, we propose RL-aware distillation (RLAD), which performs selective imitation during RL -- guiding the student toward the teacher only when it improves the current policy update. Our core component, Trust Region Ratio Distillation (TRRD), replaces the teacher-student KL regularizer with a PPO/GRPO-style likelihood-ratio objective anchored to a teacher--old-policy mixture, yielding advantage-aware, trust-region-bounded distillation on student rollouts and naturally balancing exploration, exploitation, and imitation. Across diverse logic reasoning and math benchmarks, RLAD consistently outperforms offline distillation, standard GRPO, and KL-based on-policy teacher-student knowledge distillation.