UITron-Speech: Towards Automated GUI Agents Based on Speech Instructions

TL;DR

UITron-Speech introduces an end-to-end speech-based GUI agent that processes speech instructions and screenshots, enhancing accessibility and interaction without relying on text inputs, supported by synthesized datasets and a novel training strategy.

Contribution

It is the first to enable direct speech instruction processing for GUI agents, utilizing synthesized datasets and a mixed-modality training approach to improve performance and accessibility.

Findings

Achieves robust performance across multiple benchmarks.

Demonstrates superior adaptability to speech-driven interactions.

Validates the effectiveness of synthesized datasets and training strategies.

Abstract

Autonomous agents for Graphical User Interfaces (GUIs) are revolutionizing human-computer interaction, yet their reliance on text-based instructions imposes limitations on accessibility and convenience, particularly in hands-free scenarios. To address this issue, we propose replacing text with speech as the instruction input modality for GUI agents, and introduce UITron-Speech, which is the first end-to-end GUI agent capable of directly processing speech instructions and on-device screenshots to predict user actions. To tackle the problem of data scarcity, we synthesize high-quality speech instruction datasets using a random-speaker text-to-speech model. Additionally, we design a mixed-modality training strategy to mitigate the inherent modality imbalance in pre-trained foundation models. Furthermore, we conduct a statistical analysis of the distribution of GUI grounding prediction errors and propose a training-free two-step grounding refinement method to alleviate minor localization deviations. Extensive experiments on multiple benchmarks demonstrate that UITron-Speech achieves robust performance and superior adaptability, underscoring the feasibility and potential of speech-driven GUI agents for more accessible and intelligent human-computer interaction. Our code and datasets are available at https://github.com/UITron-hub/UITron-Speech.