SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks

TL;DR

SkillLearnBench is a new benchmark for evaluating continual skill learning methods for real-world tasks, revealing that current techniques improve over no-skill baselines but face challenges in generalization and scaling.

Contribution

The paper introduces SkillLearnBench, the first comprehensive benchmark for continual skill learning, and evaluates various methods, highlighting their strengths and limitations across diverse real-world tasks.

Findings

All continual learning methods outperform no-skill baseline.

Scaling to stronger LLMs does not reliably improve skills.

External feedback facilitates genuine improvement, self-feedback can cause drift.

Abstract

Skills have become the de facto way to enable LLM agents to perform complex real-world tasks with customized instructions, workflows, and tools, but how to learn them automatically and effectively remains unclear. We introduce SkillLearnBench, the first benchmark for evaluating continual skill learning methods, comprising 20 verified, skill-dependent tasks across 15 sub-domains derived from a real-world skill taxonomy , evaluated at three levels: skill quality, execution trajectory, and task outcome. Using this benchmark, we evaluate recent continual learning techniques, those leveraging one-shot, self/teacher feedback, and skill creator to generate skills from agent experiences. We find that all continual learning methods improve over the no-skill baseline, yet consistent gains remain elusive: no method leads across all tasks and LLMs, and scaling to stronger LLMs does not reliably help. Continual learning improves tasks with clear, reusable workflows but struggles on open-ended tasks, and using stronger LLM backbones does not consistently produce better skills. Our analysis also revealed that multiple iterations in continual learning facilitate genuine improvement via external feedback, whereas self-feedback alone induces recursive drift. Our data and code are open-source at https://github.com/cxcscmu/SkillLearnBench to enable further studies of automatic skill generation and continual learning techniques.