CUA-Skill: Develop Skills for Computer Using Agent

TL;DR

CUA-Skill introduces a structured, reusable skill library for computer-using agents, significantly enhancing their success rates and robustness in automating Windows tasks, and providing a scalable infrastructure for future development.

Contribution

The paper presents CUA-Skill, a large-scale, structured skill library for computer-using agents, enabling more reliable and scalable automation of Windows applications.

Findings

Achieves 57.5% success rate on WindowsAgentArena, outperforming prior methods.

Substantially improves robustness and execution success in end-to-end benchmarks.

Provides a scalable infrastructure for future agent development.

Abstract

Computer-Using Agents (CUAs) aim to autonomously operate computer systems to complete real-world tasks. However, existing agentic systems remain difficult to scale and lag behind human performance. A key limitation is the absence of reusable and structured skill abstractions that capture how humans interact with graphical user interfaces and how to leverage these skills. We introduce CUA-Skill, a computer-using agentic skill base that encodes human computer-use knowledge as skills coupled with parameterized execution and composition graphs. CUA-Skill is a large-scale library of carefully engineered skills spanning common Windows applications, serving as a practical infrastructure and tool substrate for scalable, reliable agent development. Built upon this skill base, we construct CUA-Skill Agent, an end-to-end computer-using agent that supports dynamic skill retrieval, argument instantiation, and memory-aware failure recovery. Our results demonstrate that CUA-Skill substantially improves execution success rates and robustness on challenging end-to-end agent benchmarks, establishing a strong foundation for future computer-using agent development. On WindowsAgentArena, CUA-Skill Agent achieves state-of-the-art 57.5% (best of three) successful rate while being significantly more efficient than prior and concurrent approaches. The project page is available at https://microsoft.github.io/cua_skill/.