Trifuse: Enhancing Attention-Based GUI Grounding via Multimodal Fusion

TL;DR

Trifuse is a novel attention-based GUI grounding framework that explicitly integrates spatial anchors and multimodal cues, achieving strong performance without fine-tuning and reducing dependence on large annotated datasets.

Contribution

The paper introduces Trifuse, a new multimodal fusion strategy that enhances attention-based GUI grounding by explicitly incorporating spatial anchors, OCR cues, and caption semantics.

Findings

Achieves strong performance on four benchmarks without fine-tuning.

Incorporating OCR and caption cues improves grounding accuracy.

Reduces reliance on large annotated GUI datasets.

Abstract

GUI grounding maps natural language instructions to the correct interface elements, serving as the perception foundation for GUI agents. Existing approaches predominantly rely on fine-tuning multimodal large language models (MLLMs) using large-scale GUI datasets to predict target element coordinates, which is data-intensive and generalizes poorly to unseen interfaces. Recent attention-based alternatives exploit localization signals in MLLMs attention mechanisms without task-specific fine-tuning, but suffer from low reliability due to the lack of explicit and complementary spatial anchors in GUI images. To address this limitation, we propose Trifuse, an attention-based grounding framework that explicitly integrates complementary spatial anchors. Trifuse integrates attention, OCR-derived textual cues, and icon-level caption semantics via a Consensus-SinglePeak (CS) fusion strategy that enforces cross-modal agreement while retaining sharp localization peaks. Extensive evaluations on four grounding benchmarks demonstrate that Trifuse achieves strong performance without task-specific fine-tuning, substantially reducing the reliance on expensive annotated data. Moreover, ablation studies reveal that incorporating OCR and caption cues consistently improves attention-based grounding performance across different backbones, highlighting its effectiveness as a general framework for GUI grounding.