Training a Vision Language Model as Smartphone Assistant

TL;DR

This paper introduces a visual language model that enables smartphones to perform diverse user tasks by interpreting screen images and executing gestures, advancing mobile assistant capabilities.

Contribution

The work presents a novel vision-language model that interacts with mobile UI solely through visual inputs and actions, unlike prior models limited to single images or predefined tasks.

Findings

Effective on Android in the Wild benchmark

Handles diverse app interactions

Operates using sequences of screenshots and actions

Abstract

Addressing the challenge of a digital assistant capable of executing a wide array of user tasks, our research focuses on the realm of instruction-based mobile device control. We leverage recent advancements in large language models (LLMs) and present a visual language model (VLM) that can fulfill diverse tasks on mobile devices. Our model functions by interacting solely with the user interface (UI). It uses the visual input from the device screen and mimics human-like interactions, encompassing gestures such as tapping and swiping. This generality in the input and output space allows our agent to interact with any application on the device. Unlike previous methods, our model operates not only on a single screen image but on vision-language sentences created from sequences of past screenshots along with corresponding actions. Evaluating our method on the challenging Android in the Wild benchmark demonstrates its promising efficacy and potential.