R-VLM: Region-Aware Vision Language Model for Precise GUI Grounding

TL;DR

R-VLM introduces a region-aware vision language model with IoU-aware training for precise GUI element grounding, significantly improving accuracy over existing methods across multiple benchmarks.

Contribution

It presents a novel approach combining region proposals and IoU-aware loss to enhance GUI grounding accuracy in vision language models.

Findings

Achieves 13% higher grounding accuracy on GUI benchmarks.

Improves GUI navigation task accuracy by up to 9.7%.

Bridges vision language models with object detection techniques.

Abstract

Visual agent models for automating human activities on Graphical User Interfaces (GUIs) have emerged as a promising research direction, driven by advances in large Vision Language Models (VLMs). A critical challenge in GUI automation is the precise grounding of interface elements across diverse platforms. Existing vision-only GUI agents directly ground elements from large and cluttered screenshots, requiring them to process substantial irrelevant information that compromises their accuracy. In addition, these approaches typically employ basic cross-entropy loss for learning grounding objectives, which fails to effectively capture grounding quality compared to established object detection metrics like Intersection-over-Union (IoU). To address these issues, we introduce R-VLM, a novel GUI grounding approach that leverages zoomed-in region proposals for precise element localization. We also propose an IoU-aware objective function that facilitates model convergence toward high IoU predictions. Our approach bridges the gap between VLMs and conventional object detection techniques, improving the state-of-the-art grounding accuracy by 13% across diverse GUI platforms on the GUI grounding benchmarks ScreenSpot and AgentStudio. In addition, our R-VLM approach shows 3.2-9.7% absolute accuracy improvements in GUI navigation tasks on the AITW and Mind2Web benchmarks.