Mapping the Design Space of User Experience for Computer Use Agents

TL;DR

This paper explores the user experience design space for LLM-based computer use agents through a taxonomy and empirical study, helping developers create more user-centered agents.

Contribution

It introduces a comprehensive UX taxonomy for computer use agents and validates it through user studies, highlighting key design considerations.

Findings

Validated taxonomy of UX considerations for agents

Identified user needs and preferences across scenarios

Provided design insights for improving agent usability

Abstract

Large language model (LLM)-based computer use agents execute user commands by interacting with available UI elements, but little is known about how users want to interact with these agents or what design factors matter for their user experience (UX). We conducted a two-phase study to map the UX design space for computer use agents. In Phase 1, we reviewed existing systems to develop a taxonomy of UX considerations, then refined it through interviews with eight UX and AI practitioners. The resulting taxonomy included categories such as user prompts, explainability, user control, and users' mental models, with corresponding subcategories and example design features. In Phase 2, we ran a Wizard-of-Oz study with 20 participants, where a researcher acted as a web-based computer use agent and probed user reactions during normal, error-prone and risky execution. We used the findings to validate the taxonomy from Phase 1 and deepen our understand of the design space by identifying the connections between design areas and divergence in user needs and scenarios. Our taxonomy and empirical insights provide a map for developers to consider different aspects of user experience in computer use agent design and to situate their designs within users' diverse needs and scenarios.