FedGUI: Benchmarking Federated GUI Agents across Heterogeneous Platforms, Devices, and Operating Systems

TL;DR

FedGUI introduces a comprehensive benchmark for federated GUI agents across diverse platforms, enabling systematic study of heterogeneity and fostering scalable, privacy-preserving GUI solutions.

Contribution

It provides the first benchmark with datasets and experiments for federated GUI agents across multiple platforms and heterogeneity types.

Findings

Cross-platform collaboration enhances federated GUI performance.

Platform and OS are the most influential heterogeneity factors.

FedGUI facilitates development of scalable, privacy-preserving GUI agents.

Abstract

Training GUI agents with traditional centralized methods faces significant cost and scalability challenges. Federated learning (FL) offers a promising solution, yet its potential is hindered by the lack of benchmarks that capture real-world, cross-platform heterogeneity. To bridge this gap, we introduce FedGUI, the first comprehensive benchmark for developing and evaluating federated GUI agents across mobile, web, and desktop platforms. FedGUI provides a suite of six curated datasets to systematically study four crucial types of heterogeneity: cross-platform, cross-device, cross-OS, and cross-source. Extensive experiments reveal several key insights: First, we show that cross-platform collaboration improves performance, extending prior mobile-only federated learning to diverse GUI environments; Second, we demonstrate the presence of distinct heterogeneity dimensions and identify platform and OS as the most influential factors. FedGUI provides a vital foundation for the community to build more scalable and privacy-preserving GUI agents for real-world deployment. Our code and data are publicly available at https://github.com/wwh0411/FedGUI..