SkillGenBench: Benchmarking Skill Generation Pipelines for LLM Agents

TL;DR

SkillGenBench is a new benchmark designed to evaluate the ability of language models to generate correct, reusable, and executable skills from repositories and documents, addressing a key challenge in building effective LLM agents.

Contribution

It introduces a unified, controlled protocol for assessing skill generation pipelines, covering multiple generation regimes and procedural sources, with standardized evaluation procedures.

Findings

Performance varies significantly across different skill-generation methods.

Reusability and correctness of generated skills remain challenging.

Distinct failure modes are observed between repository-based and document-based skill generation.

Abstract

As LLM agents are increasingly built around reusable skills, a central challenge is no longer only whether agents can use provided skills, but whether they can generate correct, reusable, and executable skills from repositories and documents. Existing benchmarks primarily evaluate the efficacy of given skills or the ability of agents to solve downstream tasks from raw context, but they do not isolate skill generation itself as the object of study. We introduce SkillGenBench, a benchmark for evaluating skill generation pipelines under a unified and controlled protocol. In SkillGenBench, a generator receives raw corpora and produces standardized skill artifacts, which are then executed under fixed harnesses and assessed with unified evaluation procedures. The benchmark covers two generation regimes: task-conditioned generation, where a task-specific skill is synthesized after the task is revealed, and task-agnostic generation, where a reusable skill library must be distilled before downstream tasks are known. It also spans two complementary procedural sources: repository-grounded instances, where procedures are distributed across code, configuration, and scripts, and document-grounded instances, where procedures and constraints must be distilled from long-form text. We provide standardized task specifications, pinned environments, and evaluation protocols centered on deterministic execution-based checks, supplemented by auxiliary signals for diagnosis. Experiments across a range of skill-generation methods and backbones show substantial performance variation, highlight the difficulty of reusable skill distillation, and reveal distinct failure modes in skill generation from software repositories versus long-form documents. SkillGenBench establishes a reproducible testbed for studying skill generation as an independent research problem in agent systems.