Mapping the Design Space of User Experience for Computer Use Agents
Ruijia Cheng, Jenny T. Liang, Eldon Schoop, Jeffrey Nichols

TL;DR
This paper explores the user experience design space for LLM-based computer use agents through a taxonomy and empirical study, helping developers create more user-centered agents.
Contribution
It introduces a comprehensive UX taxonomy for computer use agents and validates it through user studies, highlighting key design considerations.
Findings
Validated taxonomy of UX considerations for agents
Identified user needs and preferences across scenarios
Provided design insights for improving agent usability
Abstract
Large language model (LLM)-based computer use agents execute user commands by interacting with available UI elements, but little is known about how users want to interact with these agents or what design factors matter for their user experience (UX). We conducted a two-phase study to map the UX design space for computer use agents. In Phase 1, we reviewed existing systems to develop a taxonomy of UX considerations, then refined it through interviews with eight UX and AI practitioners. The resulting taxonomy included categories such as user prompts, explainability, user control, and users' mental models, with corresponding subcategories and example design features. In Phase 2, we ran a Wizard-of-Oz study with 20 participants, where a researcher acted as a web-based computer use agent and probed user reactions during normal, error-prone and risky execution. We used the findings to…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAI in Service Interactions · Usability and User Interface Design · Social Robot Interaction and HRI
