UXAgent: An LLM Agent-Based Usability Testing Framework for Web Design

TL;DR

UXAgent leverages Large Language Model-simulated agents to automate and enhance web usability testing, providing qualitative, quantitative, and video data to assist UX researchers in refining their study designs before involving real users.

Contribution

This paper introduces UXAgent, a novel framework combining LLM-based agents and browser integration to simulate user interactions for usability testing, enabling pre-study design evaluation.

Findings

Researchers found UXAgent innovative and useful for usability testing.

Participants expressed optimism but also concerns about LLM agent-assisted UX research.

The system generated diverse simulated user interactions and data formats.

Abstract

Usability testing is a fundamental yet challenging (e.g., inflexible to iterate the study design flaws and hard to recruit study participants) research method for user experience (UX) researchers to evaluate a web design. Recent advances in Large Language Model-simulated Agent (LLM-Agent) research inspired us to design UXAgent to support UX researchers in evaluating and reiterating their usability testing study design before they conduct the real human subject study. Our system features an LLM-Agent module and a universal browser connector module so that UX researchers can automatically generate thousands of simulated users to test the target website. The results are shown in qualitative (e.g., interviewing how an agent thinks ), quantitative (e.g., # of actions), and video recording formats for UX researchers to analyze. Through a heuristic user evaluation with five UX researchers, participants praised the innovation of our system but also expressed concerns about the future of LLM Agent-assisted UX study.