Towards LLM-Based Usability Analysis for Recommender User Interfaces

TL;DR

This paper investigates using multimodal large language models to automate usability analysis of recommender system interfaces, aiming to support scalable, heuristic-style evaluations that improve user experience.

Contribution

It introduces a novel approach leveraging multimodal LLMs to assess usability of recommender UIs, demonstrating potential for scalable, automated evaluations.

Findings

LLMs can analyze UI screenshots for usability criteria

Automated assessments support heuristic evaluations at scale

Potential to enhance user experience through AI-driven analysis

Abstract

Usability is a key factor in the effectiveness of recommender systems. However, the analysis of user interfaces is a time-consuming process that requires expertise. Recent advances in multimodal large language models (LLMs) offer promising opportunities to automate such evaluations. In this work, we explore the potential of multimodal LLMs to assess the usability of recommender system interfaces by considering a variety of publicly available systems as examples. We take user interface screenshots from multiple of these recommender platforms to cover both preference elicitation and recommendation presentation scenarios. An LLM is instructed to analyze these interfaces with regard to different usability criteria and provide explanatory feedback. Our evaluation demonstrates how LLMs can support heuristic-style usability assessments at scale to support the improvement of user experience.