OS-MAP: How Far Can Computer-Using Agents Go in Breadth and Depth?

TL;DR

OS-MAP introduces a comprehensive benchmark with 416 tasks across 15 applications, evaluating computer-using agents' automation levels and generalization scope to identify current limitations and guide future development.

Contribution

The paper presents OS-MAP, a novel benchmark that systematically assesses agent capabilities across multiple dimensions, addressing gaps in existing benchmarks and aligning evaluation with real-world user demands.

Findings

State-of-the-Art agents struggle with high-level tasks involving perception and reasoning.

Current agents show limited generalization across complex demand hierarchies.

Benchmark reveals significant gaps in agent autonomy and adaptability.

Abstract

Computer-using agents have shown strong potential to boost human productivity and enable new application forms across platforms. While recent advances have led to usable applications, existing benchmarks fail to account for the internal task heterogeneity and the corresponding agent capabilities, as well as their alignment with actual user demands-hindering both targeted capability development and the reliable transition of research progress into practical deployment. To bridge the gap, we present OS-MAP, a benchmark for daily computer-using automation that organizes its 416 realistic tasks across 15 applications along two key dimensions: a five-level taxonomy of automation and a generalization scope derived from a real-world user demand hierarchy. To enable fine-grained analysis of required capabilities and alignment with real-world scenarios, OS-MAP evaluates agents along two dimensions: automation level across a five-level taxonomy, and generalization scope across a demand hierarchy. This design captures varying levels of required agent autonomy and generalization, forming a performance-generalization evaluation matrix for structured and comprehensive assessment. Experiments show that even State-of-the-Art agents with VLM backbones struggle with higher-level tasks involving perception, reasoning, and coordination-highlighting the need for a deeper understanding of current strengths and limitations to drive the future progress in computer-using agents research and deployment. All code, environments, baselines, and data are publicly available at https://github.com/OS-Copilot/OS-Map.