GUI-Shift: Enhancing VLM-Based GUI Agents through Self-supervised Reinforcement Learning

TL;DR

GUI-Shift introduces a self-supervised reinforcement learning framework that enhances vision-language models for GUI tasks by predicting GUI transitions, reducing reliance on annotated data, and improving performance across multiple benchmarks.

Contribution

The paper presents GUI-Shift, a novel self-supervised RL method that leverages GUI dynamics prediction to improve VLM-based GUI agents without requiring natural language instructions.

Findings

Up to 11.2% increase in GUI automation accuracy

Effective generalization across multiple GUI benchmarks

Scalable training without annotated datasets

Abstract

Training effective Vision-Language Models (VLMs) for GUI agents typically depends on large-scale annotated datasets, whose collection is both labor-intensive and error-prone. We introduce K-step GUI Transition, a self-supervised inverse dynamics task in which VLMs learn GUI dynamics by predicting the initial action that causes a transition between two GUI states. This approach eliminates the need for natural language instructions and enables scalable dataset construction from existing GUI trajectories or automated exploration. Building on this task, we propose GUI-Shift, a reinforcement learning (RL) framework that combines rule-based optimization with data filtering to improve VLM performance. We conduct extensive experiments using multiple VLM backbones across four benchmarks, spanning GUI task automation (AndroidControl, GUI Odyssey) and GUI grounding (ScreenSpot-v2, ScreenSpot-Pro). Our results show that training on GUI-Shift generalizes well to both GUI automation and grounding tasks, yielding up to an 11.2% increase in GUI automation accuracy. This study underscores the potential of self-supervised RL to leverage unlabeled GUI trajectories and offers a scalable alternative to training with annotated samples.