GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behavior

TL;DR

This paper introduces GUI-Reflection, a framework that enhances multimodal GUI models with self-reflection and error correction abilities through automated training stages, improving robustness and adaptability in GUI automation.

Contribution

The paper presents a novel framework integrating self-reflection into multimodal GUI models via automated data generation and online tuning, which was not addressed in prior work.

Findings

Automated data pipelines for reflection and error correction.

A new GUI-Reflection Task Suite for evaluation.

An iterative online reflection tuning algorithm.

Abstract

Multimodal Large Language Models (MLLMs) have shown great potential in revolutionizing Graphical User Interface (GUI) automation. However, existing GUI models mostly rely on learning from nearly error-free offline trajectories, thus lacking reflection and error recovery capabilities. To bridge this gap, we propose GUI-Reflection, a novel framework that explicitly integrates self-reflection and error correction capabilities into end-to-end multimodal GUI models throughout dedicated training stages: GUI-specific pre-training, offline supervised fine-tuning (SFT), and online reflection tuning. GUI-reflection enables self-reflection behavior emergence with fully automated data generation and learning processes without requiring any human annotation. Specifically, 1) we first propose scalable data pipelines to automatically construct reflection and error correction data from existing successful trajectories. While existing GUI models mainly focus on grounding and UI understanding ability, we propose the GUI-Reflection Task Suite to learn and evaluate reflection-oriented abilities explicitly. 2) Furthermore, we built a diverse and efficient environment for online training and data collection of GUI models on mobile devices. 3) We also present an iterative online reflection tuning algorithm leveraging the proposed environment, enabling the model to continuously enhance its reflection and error correction abilities. Our framework equips GUI agents with self-reflection and correction capabilities, paving the way for more robust, adaptable, and intelligent GUI automation, with all data, models, environments, and tools to be released publicly.