SkillCraft: Can LLM Agents Learn to Use Tools Skillfully?

TL;DR

SkillCraft is a new benchmark designed to evaluate and improve the ability of large language model agents to learn, compose, and reuse higher-level skills in complex, realistic tool-using scenarios, emphasizing efficiency and skill abstraction.

Contribution

The paper introduces SkillCraft, a benchmark that tests agents' ability to form and reuse compositional skills, along with a lightweight evaluation protocol for auto-assembling and caching skills.

Findings

Agents show up to 80% reduction in token usage through skill reuse.

Success correlates with the ability to compose and reuse tools.

Skill abstraction enhances efficiency and performance.

Abstract

Real-world tool-using agents operate over long-horizon workflows with recurring structure and diverse demands, where effective behavior requires not only invoking atomic tools but also abstracting, and reusing higher-level tool compositions. However, existing benchmarks mainly measure instance-level success under static tool sets, offering limited insight into agents' ability to acquire such reusable skills. We address this gap by introducing SkillCraft, a benchmark explicitly stress-test agent ability to form and reuse higher-level tool compositions, where we call Skills. SkillCraft features realistic, highly compositional tool-use scenarios with difficulty scaled along both quantitative and structural dimensions, designed to elicit skill abstraction and cross-task reuse. We further propose a lightweight evaluation protocol that enables agents to auto-compose atomic tools into executable Skills, cache and reuse them inside and across tasks, thereby improving efficiency while accumulating a persistent library of reusable skills. Evaluating state-of-the-art agents on SkillCraft, we observe substantial efficiency gains, with token usage reduced by up to 80% by skill saving and reuse. Moreover, success rate strongly correlates with tool composition ability at test time, underscoring compositional skill acquisition as a core capability.