EDGE: Enhanced Grounded GUI Understanding with Enriched Multi-Granularity Synthetic Data

TL;DR

This paper introduces EDGE, a data synthesis framework that automatically generates large-scale, multi-granularity training data from web pages to enhance GUI understanding in vision-language models, reducing manual annotation needs.

Contribution

EDGE provides a novel, automated data generation method from web pages, significantly improving GUI understanding in LVLMs without extensive manual labeling.

Findings

Models trained with EDGE data outperform baselines on GUI benchmarks.

The approach generalizes well to unseen desktop and mobile environments.

Reduces reliance on manual annotation for GUI understanding tasks.

Abstract

Autonomous agents operating on the graphical user interfaces (GUIs) of various applications hold immense practical value. Unlike the large language model (LLM)-based methods which rely on structured texts and customized backends, the approaches using large vision-language models (LVLMs) are more intuitive and adaptable as they can visually perceive and directly interact with screens, making them indispensable in general scenarios without text metadata and tailored backends. Given the lack of high-quality training data for GUI-related tasks in existing work, this paper aims to enhance the GUI understanding and interacting capabilities of LVLMs through a data-driven approach. We propose EDGE, a general data synthesis framework that automatically generates large-scale, multi-granularity training data from webpages across the Web. Evaluation results on various GUI and agent benchmarks demonstrate that the model trained with the dataset generated through EDGE exhibits superior webpage understanding capabilities, which can then be easily transferred to previously unseen desktop and mobile environments. Our approach significantly reduces the dependence on manual annotations, empowering researchers to harness the vast public resources available on the Web to advance their work. Our source code, the dataset and the model are available at https://anonymous.4open.science/r/EDGE-1CDB.