OS-ATLAS: A Foundation Action Model for Generalist GUI Agents

TL;DR

OS-ATLAS introduces a new open-source GUI action model that significantly improves GUI grounding and out-of-distribution generalization across multiple platforms, supported by a large synthesized dataset and extensive benchmarking.

Contribution

The paper presents OS-ATLAS, a foundational GUI action model with innovative data synthesis and training methods, enabling better GUI understanding and generalization in open-source VLMs.

Findings

OS-ATLAS outperforms previous models on six benchmarks.

Developed the largest open-source cross-platform GUI grounding dataset.

Demonstrated improved GUI understanding and OOD generalization.

Abstract

Existing efforts in building GUI agents heavily rely on the availability of robust commercial Vision-Language Models (VLMs) such as GPT-4o and GeminiProVision. Practitioners are often reluctant to use open-source VLMs due to their significant performance lag compared to their closed-source counterparts, particularly in GUI grounding and Out-Of-Distribution (OOD) scenarios. To facilitate future research in this area, we developed OS-Atlas - a foundational GUI action model that excels at GUI grounding and OOD agentic tasks through innovations in both data and modeling. We have invested significant engineering effort in developing an open-source toolkit for synthesizing GUI grounding data across multiple platforms, including Windows, Linux, MacOS, Android, and the web. Leveraging this toolkit, we are releasing the largest open-source cross-platform GUI grounding corpus to date, which contains over 13 million GUI elements. This dataset, combined with innovations in model training, provides a solid foundation for OS-Atlas to understand GUI screenshots and generalize to unseen interfaces. Through extensive evaluation across six benchmarks spanning three different platforms (mobile, desktop, and web), OS-Atlas demonstrates significant performance improvements over previous state-of-the-art models. Our evaluation also uncovers valuable insights into continuously improving and scaling the agentic capabilities of open-source VLMs.