Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents

TL;DR

Skill-SD introduces a dynamic self-distillation framework that transforms an agent's trajectories into natural language skills, enhancing multi-turn LLM agent training by providing adaptive supervision and stabilizing learning.

Contribution

The paper proposes Skill-SD, a novel method that uses agent-generated skills as dynamic privileged information for improved training stability and performance in multi-turn LLM agents.

Findings

Skill-SD outperforms standard RL and OPSD baselines on agentic benchmarks.

It achieves +14.0%/+10.9% improvements on AppWorld/Sokoban with GRPO.

It achieves +42.1%/+40.6% improvements with vanilla OPD.

Abstract

Reinforcement learning (RL) has been widely used to train LLM agents for multi-turn interactive tasks, but its sample efficiency is severely limited by sparse rewards and long horizons. On-policy self-distillation (OPSD) alleviates this by providing dense token-level supervision from a privileged teacher that has access to ground-truth answers. However, such fixed privileged information cannot capture the diverse valid strategies in agent tasks, and naively combining OPSD with RL often leads to training collapse. To address these limitations, we introduce Skill-SD, a framework that turns the agent's own trajectories into dynamic training-only supervision. Completed trajectories are summarized into compact natural language skills that describe successful behaviors, mistakes, and workflows. These skills serve as dynamic privileged information conditioning only the teacher, while the student always acts under the plain task prompt and learns to internalize the guidance through distillation. To stabilize the training, we derive an importance-weighted reverse-KL loss to provide gradient-correct token-level distillation, and dynamically synchronize the teacher with the improving student. Experimental results on agentic benchmarks demonstrate that Skill-SD substantially outperforms the standard RL baseline, improving both vanilla GRPO (+14.0%/+10.9% on AppWorld/Sokoban) and vanilla OPD (+42.1%/+40.6%). Project page: https://k1xe.github.io/skill-sd/