GUI-Perturbed: Domain Randomization Reveals Systematic Brittleness in GUI Grounding Models
Yangyue Wang, Harshvardhan Sikka, Yash Mathur, Tony Zhou, Jinu Nyachhyon, Pranav Guruprasad

TL;DR
GUI-Perturbed introduces a framework to evaluate GUI grounding models' robustness by systematically perturbing visual scenes and instructions, revealing vulnerabilities not captured by standard benchmarks.
Contribution
It provides a controlled perturbation framework, GUI-Perturbed, to diagnose specific weaknesses in GUI grounding models, including spatial reasoning and visual robustness.
Findings
Relational instructions cause systematic accuracy drops across models.
A 70% browser zoom significantly degrades model performance.
Fine-tuning with augmented data can worsen model accuracy.
Abstract
GUI grounding models report over 85% accuracy on standard benchmarks, yet drop 27-56 percentage points when instructions require spatial reasoning rather than direct element naming. Current benchmarks miss this because they evaluate each screenshot once with a single fixed instruction. We introduce GUI-Perturbed, a controlled perturbation framework that independently varies visual scenes and instructions to measure grounding robustness. Evaluating three 7B models from the same architecture lineage, we find that relational instructions cause systematic accuracy collapse across all models, a 70% browser zoom produces statistically significant degradation, and rank-8 LoRA fine-tuning with augmented data degrades performance rather than improving it. By perturbing along independent axes, GUI-Perturbed isolates which specific capability axes are affected-spatial reasoning, visual robustness,…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
