InquireMobile: Teaching VLM-based Mobile Agent to Request Human Assistance via Reinforcement Fine-Tuning

TL;DR

InquireMobile is a reinforcement learning-based mobile agent model that actively requests human assistance at critical points, significantly improving safe interaction capabilities in real-world environments.

Contribution

The paper introduces InquireMobile, a novel model with a two-stage training strategy and pre-action reasoning, enhancing proactive inquiry and safety in VLM-based mobile agents.

Findings

46.8% improvement in inquiry success rate

Achieves best overall success rate on InquireBench

Open-sources datasets, models, and evaluation code

Abstract

Recent advances in Vision-Language Models (VLMs) have enabled mobile agents to perceive and interact with real-world mobile environments based on human instructions. However, the current fully autonomous paradigm poses potential safety risks when model understanding or reasoning capabilities are insufficient. To address this challenge, we first introduce \textbf{InquireBench}, a comprehensive benchmark specifically designed to evaluate mobile agents' capabilities in safe interaction and proactive inquiry with users, encompassing 5 categories and 22 sub-categories, where most existing VLM-based agents demonstrate near-zero performance. In this paper, we aim to develop an interactive system that actively seeks human confirmation at critical decision points. To achieve this, we propose \textbf{InquireMobile}, a novel model inspired by reinforcement learning, featuring a two-stage training strategy and an interactive pre-action reasoning mechanism. Finally, our model achieves an 46.8% improvement in inquiry success rate and the best overall success rate among existing baselines on InquireBench. We will open-source all datasets, models, and evaluation codes to facilitate development in both academia and industry.