MEGA-GUI: Multi-stage Enhanced Grounding Agents for GUI Elements

TL;DR

MEGA-GUI introduces a multi-stage, modular framework for GUI element grounding that significantly improves accuracy over monolithic models by effectively handling visual clutter and semantic ambiguity.

Contribution

It proposes a novel multi-stage approach with specialized agents, including a bidirectional ROI zoom and context-aware rewriting, enhancing GUI grounding performance.

Findings

Achieves 73.18% accuracy on ScreenSpot-Pro benchmark.

Reaches 68.63% accuracy on OSWorld-G benchmark.

Outperforms previous monolithic models in GUI grounding tasks.

Abstract

Graphical User Interface (GUI) grounding - the task of mapping natural language instructions to screen coordinates - is essential for autonomous agents and accessibility technologies. Existing systems rely on monolithic models or one-shot pipelines that lack modularity and fail under visual clutter and ambiguous instructions. We introduce MEGA-GUI, a multi-stage framework that separates grounding into coarse Region-of-Interest (ROI) selection and fine-grained element grounding, orchestrated by specialized vision-language agents. MEGA-GUI features a bidirectional ROI zoom algorithm that mitigates spatial dilution and a context-aware rewriting agent that reduces semantic ambiguity. Our analysis reveals complementary strengths and weaknesses across vision-language models at different visual scales, and we show that leveraging this modular structure achieves consistently higher accuracy than monolithic approaches. On the visually dense ScreenSpot-Pro benchmark, MEGA-GUI attains 73.18% accuracy, and on the semantically complex OSWorld-G benchmark it reaches 68.63%, surpassing previously reported results. Code and the Grounding Benchmark Toolkit (GBT) are available at https://github.com/samsungsds-research-papers/mega-gui.