GUIDE: A Benchmark for Understanding and Assisting Users in Open-Ended GUI Tasks

TL;DR

GUIDE introduces a comprehensive benchmark dataset for evaluating AI models' ability to understand user behavior, infer intent, and assist in open-ended GUI tasks across various software, highlighting current model limitations.

Contribution

This paper presents GUIDE, a novel benchmark with annotated GUI user demonstrations, to evaluate AI's understanding of user intent and ability to assist in complex, open-ended tasks.

Findings

All models struggled with behavior state and help prediction accuracy.

Providing user context significantly improved help prediction performance.

Structured user understanding is critical for effective GUI assistance.

Abstract

Graphical User Interface (GUI) agents have the potential to assist users in interacting with complex software (e.g., PowerPoint, Photoshop). While prior research has primarily focused on automating user actions through clicks and keystrokes, this paradigm overlooks human intention, where users value the ability to explore, iterate, and refine their ideas while maintaining agency. To move beyond automation and toward collaboration, GUI agents must understand what users are doing and why. We introduce GUIDE (GUI User Intent Detection Evaluation), a benchmark that evaluates AI models on their ability to perceive user behavior, infer intent, and provide assistance in open-ended GUI tasks. GUIDE consists of 67.5 hours of screen recordings from 120 novice user demonstrations with think-aloud narrations, across 10 software. GUIDE defines three tasks - (i) Behavior State Detection, (ii) Intent Prediction, and (iii) Help Prediction that test a model's ability to recognize behavior state, reason about goals, and decide when and how to help. Evaluations across eight state-of-the-art multimodal models reveal that all models struggled, achieving only 44.6% and 55.0% accuracy on behavior state and help prediction. However, providing user context significantly improved the performance, raising help prediction by up to 50.2pp, highlighting the critical role of structured user understanding in effective assistance. Our dataset is available at https://guide-bench.github.io.