Self-Distilled Agentic Reinforcement Learning

TL;DR

SDAR introduces a gated auxiliary objective for reinforcement learning agents, enhancing stability and performance by integrating dense token-level guidance from self-distillation, especially in multi-turn scenarios.

Contribution

It proposes SDAR, a novel method that combines self-distillation with RL using a gating mechanism to improve multi-turn agent training stability and effectiveness.

Findings

SDAR outperforms baseline methods on ALFWorld, WebShop, and Search-QA datasets.

SDAR achieves +9.4% on ALFWorld and +10.2% on WebShop-Acc.

SDAR avoids instability issues present in naive GRPO+OPSD approaches.

Abstract

Reinforcement learning (RL) has emerged as a central paradigm for post-training LLM agents, yet its trajectory-level reward signal provides only coarse supervision for long-horizon interaction. On-Policy Self-Distillation (OPSD) complements RL by introducing dense token-level guidance from a teacher branch augmented with privileged context. However, transferring OPSD to multi-turn agents proves problematic: compounding multi-turn instability destabilizes supervision, while skill-conditioned privileged guidance requires asymmetric treatment for negative teacher rejections may arise from imperfect skills retrieval or utilization. We introduce SDAR (Self-Distilled Agentic Reinforcement Learning), which treats OPSD as a gated auxiliary objective while keeping RL as the primary optimization backbone. SDAR maps detached token-level signals into a sigmoid gate, strengthening distillation on teacher-endorsed positive-gap tokens and softly attenuating negative teacher rejections. Across the Qwen2.5 and Qwen3 families on ALFWorld, WebShop, and Search-QA, SDAR substantially improves over GRPO (+9.4% on ALFWorld, +7.0% on Search-QA, +10.2% on WebShop-Acc), avoids the instability of naive GRPO+OPSD, and consistently outperforms hybrid RL--OPSD baselines across model scales.