SkillGraph: Skill-Augmented Reinforcement Learning for Agents via Evolving Skill Graphs

TL;DR

This paper introduces SKILLGRAPH, a dynamic skill graph framework that enhances reinforcement learning agents by structuring and evolving skill dependencies to improve task composition and performance.

Contribution

It presents a novel skill graph representation with typed edges, enabling better skill reuse, dependency modeling, and continuous improvement through reinforcement learning feedback.

Findings

Achieves state-of-the-art performance on multiple tasks.

Significantly improves performance on complex, multi-step tasks.

Demonstrates effective skill dependency modeling and dynamic library updates.

Abstract

Skill libraries enable large language model agents to reuse experience from past interactions, but most existing libraries store skills as isolated entries and retrieve them only by semantic similarity. This leads to two key challenges for compositional tasks. Firstly, an agent must identify not only relevant skills but also how they depend on and build upon each other. Secondly, it also makes library maintenance difficult, since the system lacks structural cues for deciding when skills should be merged, split, or removed. We propose SKILLGRAPH, a framework that represents reusable skills as nodes in a directed graph, with typed edges encoding prerequisite, enhancement, and co-occurrence relations. Given a new task, SKILLGRAPH retrieves not just individual skills, but an ordered skill subgraph that can guide multi-step decision making. The graph is continuously updated from agent trajectories and reinforcement learning feedback, allowing both the skill library and the agent policy to improve together. Experiments on ALFWorld, WebShop, and seven search-augmented QA tasks show that SKILLGRAPH achieves state-of-the-art performance against memory-augmented RL methods, with especially large gains on complex tasks that require composing multiple skills.