Test-Time Reinforcement Learning for GUI Grounding via Region Consistency

TL;DR

This paper introduces GUI-RC and GUI-RCPO, novel test-time methods that improve GUI grounding accuracy by leveraging spatial consensus among multiple predictions, reducing reliance on labeled data.

Contribution

It presents the first test-time scaling and reinforcement learning techniques for GUI grounding, enhancing accuracy without additional training data.

Findings

GUI-RC improves accuracy by 2-3% without training.

GUI-RCPO achieves 3-6% accuracy gains using unlabeled data.

The methods are effective across various architectures.

Abstract

Graphical User Interface (GUI) grounding, the task of mapping natural language instructions to precise screen coordinates, is fundamental to autonomous GUI agents. While existing methods achieve strong performance through extensive supervised training or reinforcement learning with labeled rewards, they remain constrained by the cost and availability of pixel-level annotations. We observe that when models generate multiple predictions for the same GUI element, the spatial overlap patterns reveal implicit confidence signals that can guide more accurate localization. Leveraging this insight, we propose GUI-RC (Region Consistency), a test-time scaling method that constructs spatial voting grids from multiple sampled predictions to identify consensus regions where models show highest agreement. Without any training, GUI-RC improves accuracy by 2-3% across various architectures on ScreenSpot benchmarks. We further introduce GUI-RCPO (Region Consistency Policy Optimization), transforming these consistency patterns into rewards for test-time reinforcement learning. By computing how well each prediction aligns with the collective consensus, GUI-RCPO enables models to iteratively refine their outputs on unlabeled data during inference. Extensive experiments demonstrate the generality of our approach: using only 1,272 unlabeled data, GUI-RCPO achieves 3-6% accuracy improvements across various architectures on ScreenSpot benchmarks. Our approach reveals the untapped potential of test-time scaling and test-time reinforcement learning for GUI grounding, offering a promising path toward more data-efficient GUI agents.