A Comprehensive Survey of Agents for Computer Use: Foundations, Challenges, and Future Directions

TL;DR

This survey comprehensively reviews agents for computer use (ACUs), analyzing their current state, challenges, and future directions, with a focus on taxonomy, research gaps, and practical deployment considerations.

Contribution

It introduces a unifying taxonomy for ACUs, reviews 87 systems and 33 datasets, and identifies key research gaps and future directions for practical, general-purpose ACUs.

Findings

Identified six major research gaps including generalization and evaluation issues.

Reviewed 87 ACUs and 33 datasets across different approaches.

Proposed strategies to improve ACU robustness and real-world applicability.

Abstract

Agents for computer use (ACUs) are an emerging class of systems capable of executing complex tasks on digital devices -- such as desktops, mobile phones, and web platforms -- given instructions in natural language. These agents can automate tasks by controlling software via low-level actions like mouse clicks and touchscreen gestures. However, despite rapid progress, ACUs are not yet mature for everyday use. In this survey, we investigate the state-of-the-art, trends, and research gaps in the development of practical ACUs. We provide a comprehensive review of the ACU landscape, introducing a unifying taxonomy spanning three dimensions: (I) the domain perspective, characterizing agent operating contexts; (II) the interaction perspective, describing observation modalities (e.g., screenshots, HTML) and action modalities (e.g., mouse, keyboard, code execution); and (III) the agent perspective, detailing how agents perceive, reason, and learn. We review 87 ACUs and 33 datasets across foundation model-based and classical approaches through this taxonomy. Our analysis identifies six major research gaps: insufficient generalization, inefficient learning, limited planning, low task complexity in benchmarks, non-standardized evaluation, and a disconnect between research and practical conditions. To address these gaps, we advocate for: (a) vision-based observations and low-level control to enhance generalization; (b) adaptive learning beyond static prompting; (c) effective planning and reasoning methods and models; (d) benchmarks that reflect real-world task complexity; (e) standardized evaluation based on task success; (f) aligning agent design with real-world deployment constraints. Together, our taxonomy and analysis establish a foundation for advancing ACU research toward general-purpose agents for robust and scalable computer use.