GUI Agents: A Survey

TL;DR

This survey reviews the development, architectures, and evaluation of GUI agents powered by large foundation models, highlighting their capabilities, challenges, and future research directions in automating human-computer interaction.

Contribution

It provides a comprehensive categorization and unified framework for GUI agents, summarizing current progress, benchmarks, and open challenges in the field.

Findings

Categorization of benchmarks and evaluation metrics

Unified framework for perception, reasoning, and acting

Identification of open challenges and future directions

Abstract

Graphical User Interface (GUI) agents, powered by Large Foundation Models, have emerged as a transformative approach to automating human-computer interaction. These agents autonomously interact with digital systems or software applications via GUIs, emulating human actions such as clicking, typing, and navigating visual elements across diverse platforms. Motivated by the growing interest and fundamental importance of GUI agents, we provide a comprehensive survey that categorizes their benchmarks, evaluation metrics, architectures, and training methods. We propose a unified framework that delineates their perception, reasoning, planning, and acting capabilities. Furthermore, we identify important open challenges and discuss key future directions. Finally, this work serves as a basis for practitioners and researchers to gain an intuitive understanding of current progress, techniques, benchmarks, and critical open problems that remain to be addressed.