KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learning

TL;DR

KDRL introduces a unified post-training framework combining knowledge distillation and reinforcement learning to enhance reasoning capabilities of large language models, achieving better performance and efficiency on reasoning benchmarks.

Contribution

This work presents the first unified framework that jointly optimizes reasoning LLMs through teacher supervision and self-exploration, integrating KD and RL with a systematic analysis of their interactions.

Findings

KDRL outperforms existing methods on multiple reasoning benchmarks.

The unified approach balances reasoning performance and token efficiency.

Different KL approximations and reward strategies significantly influence training dynamics.

Abstract

Recent advances in large language model (LLM) post-training have leveraged two distinct paradigms to enhance reasoning capabilities: reinforcement learning (RL) and knowledge distillation (KD). While RL enables the emergence of complex reasoning behaviors, it often suffers from low sample efficiency when the initial policy struggles to explore high-reward trajectories. Conversely, KD improves learning efficiency via mimicking the teacher model but tends to generalize poorly to out-of-domain scenarios. In this work, we present \textbf{KDRL}, a \textit{unified post-training framework} that jointly optimizes a reasoning model through teacher supervision (KD) and self-exploration (RL). Specifically, KDRL leverages policy gradient optimization to simultaneously minimize the reverse Kullback-Leibler divergence (RKL) between the student and teacher distributions while maximizing the expected rule-based rewards. We first formulate a unified objective that integrates GRPO and KD, and systematically explore how different KL approximations, KL coefficients, and reward-guided KD strategies affect the overall post-training dynamics and performance. Empirical results on multiple reasoning benchmarks demonstrate that KDRL outperforms GRPO and various KD baselines while achieving a favorable balance between performance and reasoning token efficiency. These findings indicate that integrating KD and RL serves as an effective and efficient strategy to train reasoning LLMs.