User Reviews as a Source for Usability Requirements: A Precursor Study on Using Large Language Models

TL;DR

This study investigates whether large language models can effectively analyze user reviews for usability requirements, offering a quick, cost-effective alternative to human analysis with promising initial results.

Contribution

It introduces a dataset of user reviews labeled by humans and LLMs, develops a prompt for usability detection, and evaluates LLM performance compared to human raters.

Findings

LLMs can recognize usability as a non-functional requirement in reviews.

Performance of LLMs depends heavily on prompt quality.

A dataset of 300 labeled user reviews is provided.

Abstract

It is known that user-centered approaches to requirements engineering in general lead to a better suited product for the end-users. LLM4RE provides promising approaches to support the requirements elicitation process (e.g. classification of requirements). Previous approaches focus on Machine-Learning (ML) or Deep-Learning (DL) aspects, which require intensive training with a large amount of manually labeled data. LLMs, on the other hand, are pre-trained on large amounts of user-generated text data, enabling a user-centric workflow to analyze requirements. In this paper, we explore the possibility of exploiting the improved natural language understanding of LLMs, rather than strict ML classification, together with the mass extraction of user reviews to analyze if the performance of LLMs in understanding user reviews is comparable to the performance of human raters. This enables a quick and cheap workflow for development teams to gather and process their user\'s requirements. This paper provides three major contributions: (1) We provide a completely coded dataset of 300 user reviews containing usability-relevant aspects from three different types of apps, that were labeled by two human raters and by an LLM. (2) We build an initial prompt, based on two prompt engineering iterations and specifically developed coding guidelines derived from the 10 Nielsen Usability Heuristics, for LLMs to filter usability relevant user reviews. (3) We determine that LLMs are generally able to recognize usability as a non-functional requirement in user reviews, in terms of their F-score, but the performance and reliability is strongly dependent on the prompt.