Building Autonomous GUI Navigation via Agentic-Q Estimation and Step-Wise Policy Optimization

TL;DR

This paper introduces a novel framework for GUI navigation using agentic-Q estimation and step-wise policy optimization, enabling efficient, stable learning in non-stationary environments and improving performance of multimodal large language model-based agents.

Contribution

The paper presents a new MLLM-centered framework with agentic-Q estimation and step-wise policy optimization, reducing data costs and enhancing GUI navigation capabilities.

Findings

Achieves state-of-the-art performance on GUI navigation benchmarks.

Enables stable and efficient policy updates decoupled from environment interactions.

Surpasses larger-scale models in GUI grounding tasks.

Abstract

Recent advances in Multimodal Large Language Models (MLLMs) have substantially driven the progress of autonomous agents for Graphical User Interface (GUI). Nevertheless, in real-world applications, GUI agents are often faced with non-stationary environments, leading to high computational costs for data curation and policy optimization. In this report, we introduce a novel MLLM-centered framework for GUI agents, which consists of two components: agentic-Q estimation and step-wise policy optimization. The former one aims to optimize a Q-model that can generate step-wise values to evaluate the contribution of a given action to task completion. The latter one takes step-wise samples from the state-action trajectory as inputs, and optimizes the policy via reinforcement learning with our agentic-Q model. It should be noticed that (i) all state-action trajectories are produced by the policy itself, so that the data collection costs are manageable; (ii) the policy update is decoupled from the environment, ensuring stable and efficient optimization. Empirical evaluations show that our framework endows Ovis2.5-9B with powerful GUI interaction capabilities, achieving remarkable performances on GUI navigation and grounding benchmarks and even surpassing contenders with larger scales.