WinDeskGround: A Benchmark for Robust GUI Grounding in Complex Multi-Window Desktop Environments

TL;DR

WinDeskGround introduces a benchmark and framework for evaluating the robustness of GUI grounding in complex multi-window desktop environments, addressing real-world challenges like occlusion and clutter.

Contribution

It presents a novel, parametric synthesis framework and a diverse dataset to evaluate and improve GUI grounding robustness in realistic desktop scenarios.

Findings

Top-tier MLLMs perform well in simple settings but struggle with occlusion.

WinDeskGround reveals significant accuracy drops under partial occlusion.

The benchmark facilitates assessing and advancing GUI agent robustness.

Abstract

Multimodal Large Language Models (MLLMs) have revolutionized GUI automation, yet their efficacy is largely established on idealized, single-layer interfaces. This paper identifies a critical reliability gap: state-of-the-art agents face distinct robustness challenges in real-world desktop environments characterized by multi-window stacking, occlusion, and visual clutter. To address this, we introduce WinDeskGround, a novel benchmark and synthesis framework tailored for evaluating GUI grounding robustness. Unlike static datasets, our framework parametrically generates complex desktop scenarios by controlling window occlusion, layout density, and semantic similarity, thereby simulating the distribution shifts of authentic workflows. We construct a diverse meta-dataset of 1,356 high-fidelity instruction-target pairs and conduct comprehensive evaluations of five leading MLLMs. Our results demonstrate that while top-tier agents excel in simplified settings, their accuracy declines under partial occlusion. WinDeskGround provides a valuable benchmark to facilitate the assessment and advancement of GUI agent robustness in realistic environments. The code is available at https://github.com/ZZZhr-1/WinDeskGround.