SkillMaster: Toward Autonomous Skill Mastery in LLM Agents

TL;DR

SkillMaster introduces a training framework enabling LLM agents to autonomously create, refine, and select skills through experience, enhancing their ability to develop and adapt skills during complex task solving.

Contribution

The paper presents a novel autonomous skill mastery framework for LLM agents, incorporating trajectory-informed skill review, counterfactual utility evaluation, and a new training algorithm.

Findings

SkillMaster improves success rates by over 8% on ALFWorld and WebShop.

Agents trained with SkillMaster can identify and refine skill failures.

The framework enables agents to transfer skill improvements to new tasks with limited edits.

Abstract

Skills provide an effective mechanism for improving LLM agents on complex tasks, yet in existing agent frameworks, their creation, refinement, and selection are typically governed by external teachers, hand-designed rules, or auxiliary modules. As a result, skills remain external resources to be invoked, rather than capabilities that agents can develop, adapt, and internalize through experience. To endow LLM agents with autonomous skill mastery, we propose SkillMaster, a training framework that teaches agents to create new skills, refine existing skills, and select accumulated skills during task solving. This capability is achieved through three key designs. First, we train agents through trajectory-informed skill review, teaching agents to propose, update, or retain skills based on evidence from completed episodes. Second, each candidate skill edit is designed to be evaluated by its counterfactual utility on related probe tasks, providing a direct learning signal for training skill-editing decisions. Third, we introduce DualAdv-GRPO, which separately estimates advantages for task-solving actions and skill-editing decisions, stabilizing joint training across task solving and skill management. Experiments on ALFWorld and WebShop show that SkillMaster improves the overall success rate over state-of-the-art baselines by 8.8% and 9.3%, respectively, achieving the best performance among all compared methods. Further analysis reveals a marked shift in agent capability: agents trained with SkillMaster can identify skill failures, refine procedural knowledge from trajectory evidence, and transfer improvements to future tasks with limited skill-bank edits. Overall, SkillMaster moves LLM agents beyond mere skill use toward self-improving agents capable of developing, adapting, and applying their own skill repertoires.