Investigating Multimodal Large Language Models to Support Usability Evaluation

TL;DR

This paper explores how multimodal large language models can assist usability evaluation by identifying and prioritizing UI issues, offering a resource-efficient alternative to expert-driven methods.

Contribution

It introduces a framework for using MLLMs to analyze UI issues, compares their performance with experts, and presents an interactive tool for validation.

Findings

MLLMs can identify usability issues effectively.

MLLMs support prioritization of critical UI problems.

An interactive visualization tool enhances review and validation.

Abstract

Usability evaluation is an essential method to support the design of effective and intuitive user interfaces (UIs). However, it commonly relies on resource-intensive, expert-driven methods, which limit its accessibility, especially for small organizations. Recent multimodal large language models (MLLMs) have the potential to support usability evaluation by analyzing textual instructions together with visual UI context. This paper investigates the use of MLLMs as assistive tools for usability evaluation by framing the task as a prioritization problem. It identifies and explains usability issues and ranks them by severity. We report a study that compares the evaluations generated by multiple MLLMs with assessments from usability experts. The results demonstrate that MLLMs can offer complementary insights and support the efficient prioritization of critical issues. Additionally, we present an interactive visualization tool that enables the transparent review and validation of model-generated findings. Based on this, we outline concepts for integrating MLLM-based usability evaluation into real-world development workflows.