Training Computer Use Agents to Assess the Usability of Graphical User Interfaces

TL;DR

This paper introduces uxCUA, a machine learning-based computer use agent trained to assess GUI usability by simulating human-like interactions and predicting usability scores, improving accuracy over larger models.

Contribution

The work presents a novel algorithm for training CUAs to evaluate GUI usability using a large dataset, prioritizing key interaction flows and providing realistic critiques.

Findings

uxCUA outperforms larger models in usability assessment accuracy.

uxCUA produces realistic critiques of synthetic and real UIs.

The method offers a data-driven foundation for automated usability evaluation.

Abstract

Usability testing with experts and potential users can assess the effectiveness, efficiency, and user satisfaction of graphical user interfaces (GUIs) but doing so remains a costly and time-intensive process. Prior work has used computer use agents (CUAs) and other generative agents that can simulate user interactions and preference, but we show that agents still struggle to provide accurate usability assessments. In this work, we present a novel machine learning method that operationalizes a computational definition of usability to train CUAs to assess GUI usability by i) prioritizing important interaction flows, ii) executing them through human-like interactions, and iii) predicting a learned numerical usability score. We train a computer use agent, uxCUA, with our algorithm on a large-scale dataset of fully interactive user interfaces (UIs) paired with usability labels and human preferences. We show that uxCUA outperforms larger models in accurate usability assessments and produces realistic critiques of both synthetic and real UIs. More broadly, our work aims to build a principled, data-driven foundation for automated usability assessment in HCI.