Does GenAI Make Usability Testing Obsolete?

TL;DR

This paper introduces UX-LLM, a Large Vision-Language Model tool that predicts usability issues in iOS apps, offering a supplementary approach to traditional usability testing, especially beneficial for small teams with limited resources.

Contribution

The paper presents UX-LLM, a novel vision-language model-based tool for predicting usability issues, evaluated against traditional methods, highlighting its strengths and limitations.

Findings

UX-LLM achieved precision of 0.61-0.66 and recall of 0.35-0.38.

UX-LLM identified unknown usability issues in a real app.

Developers found UX-LLM useful but with integration concerns.

Abstract

Ensuring usability is crucial for the success of mobile apps. Usability issues can compromise user experience and negatively impact the perceived app quality. This paper presents UX-LLM, a novel tool powered by a Large Vision-Language Model that predicts usability issues in iOS apps. To evaluate the performance of UX-LLM, we predicted usability issues in two open-source apps of a medium complexity and asked two usability experts to assess the predictions. We also performed traditional usability testing and expert review for both apps and compared the results to those of UX-LLM. UX-LLM demonstrated precision ranging from 0.61 and 0.66 and recall between 0.35 and 0.38, indicating its ability to identify valid usability issues, yet failing to capture the majority of issues. Finally, we conducted a focus group with an app development team of a capstone project developing a transit app for visually impaired persons. The focus group expressed positive perceptions of UX-LLM as it identified unknown usability issues in their app. However, they also raised concerns about its integration into the development workflow, suggesting potential improvements. Our results show that UX-LLM cannot fully replace traditional usability evaluation methods but serves as a valuable supplement particularly for small teams with limited resources, to identify issues in less common user paths, due to its ability to inspect the source code.