VenusBench-GD: A Comprehensive Multi-Platform GUI Benchmark for Diverse Grounding Tasks

TL;DR

VenusBench-GD is a large-scale, bilingual, multi-platform GUI benchmark with hierarchical evaluation for diverse grounding tasks, revealing strengths and weaknesses of current models across basic and advanced GUI grounding challenges.

Contribution

It introduces a comprehensive, cross-platform GUI benchmark with a hierarchical taxonomy and improved annotation quality, addressing limitations of existing benchmarks.

Findings

Multimodal models now outperform specialized GUI models on basic tasks.

Advanced tasks still favor GUI-specialized models but show overfitting and poor robustness.

The benchmark enables more thorough evaluation of GUI grounding models.

Abstract

GUI grounding is a critical component in building capable GUI agents. However, existing grounding benchmarks suffer from significant limitations: they either provide insufficient data volume and narrow domain coverage, or focus excessively on a single platform and require highly specialized domain knowledge. In this work, we present VenusBench-GD, a comprehensive, bilingual benchmark for GUI grounding that spans multiple platforms, enabling hierarchical evaluation for real-word applications. VenusBench-GD contributes as follows: (i) we introduce a large-scale, cross-platform benchmark with extensive coverage of applications, diverse UI elements, and rich annotated data, (ii) we establish a high-quality data construction pipeline for grounding tasks, achieving higher annotation accuracy than existing benchmarks, and (iii) we extend the scope of element grounding by proposing a hierarchical task taxonomy that divides grounding into basic and advanced categories, encompassing six distinct subtasks designed to evaluate models from complementary perspectives. Our experimental findings reveal critical insights: general-purpose multimodal models now match or even surpass specialized GUI models on basic grounding tasks. In contrast, advanced tasks, still favor GUI-specialized models, though they exhibit significant overfitting and poor robustness. These results underscore the necessity of comprehensive, multi-tiered evaluation frameworks.