Understanding GUI Agent Localization Biases through Logit Sharpness

TL;DR

This paper introduces a detailed evaluation framework and new metrics for GUI agents based on multimodal large language models, addressing localization errors and improving model robustness and interpretability.

Contribution

It presents a novel fine-grained evaluation method, the Peak Sharpness Score metric, and a context-aware cropping technique to enhance GUI agent performance.

Findings

The framework reveals nuanced failure modes of GUI agents.

PSS effectively quantifies model uncertainty in localization.

Context-aware cropping improves model robustness without additional training.

Abstract

Multimodal large language models (MLLMs) have enabled GUI agents to interact with operating systems by grounding language into spatial actions. Despite their promising performance, these models frequently exhibit hallucinations-systematic localization errors that compromise reliability. We propose a fine-grained evaluation framework that categorizes model predictions into four distinct types, revealing nuanced failure modes beyond traditional accuracy metrics. To better quantify model uncertainty, we introduce the Peak Sharpness Score (PSS), a metric that evaluates the alignment between semantic continuity and logits distribution in coordinate prediction. Building on this insight, we further propose Context-Aware Cropping, a training-free technique that improves model performance by adaptively refining input context. Extensive experiments demonstrate that our framework and methods provide actionable insights and enhance the interpretability and robustness of GUI agent behavior.