Skill-Pro: Learning Reusable Skills from Experience via Non-Parametric PPO for LLM Agents

TL;DR

Skill-Pro enables LLM agents to autonomously learn and reuse procedural skills from experience, improving efficiency and stability without parameter updates through a novel non-parametric reinforcement learning approach.

Contribution

The paper introduces Skill-Pro, a framework for autonomous skill learning and reuse in LLM agents using a non-parametric PPO method without parameter updates.

Findings

Skill-Pro achieves higher skill reuse rates across tasks and agents.

It significantly improves performance with extreme memory compression.

Visualizations show transparent accumulation and refinement of skills.

Abstract

LLM-driven agents demonstrate strong performance in sequential decision-making but often rely on on-the-fly reasoning, re-deriving solutions even in recurring scenarios. This insufficient experience reuse leads to computational redundancy and execution instability. To bridge this gap, we propose Skill-Pro, a framework that enables agents to autonomously learn reusable procedural skills from interaction experiences without parameter updates. By formalizing a Skill-MDP, Skill-Pro transforms passive episodic narratives into executable Skills defined by activation, execution, and termination conditions to ensure executability. To achieve reliable reusability without capability degradation, we introduce Non-Parametric PPO, which leverages semantic gradients for high-quality candidate generation and a PPO Gate for robust Skill verification. Through score-based maintenance, Skill-Pro sustains compact, high-quality procedural memory. Experimental results across in-domain, cross-task, and cross-agent scenarios demonstrate that Skill-Pro achieves superior reuse rates and significant performance gains with extreme memory compression. Visualized evolutionary trajectories and Skill distributions further reveal how Skill-Pro transparently accumulates, refines, and reuses procedural knowledge to facilitate long-term autonomy.