Classification of Quality Characteristics in Online User Feedback using Linguistic Analysis, Crowdsourcing and LLMs

TL;DR

This study evaluates linguistic patterns, crowdsourcing, and large language models for classifying quality features in online user feedback, finding crowdsourcing and LLMs effective in low-data scenarios for software quality assessment.

Contribution

It compares three low-data approaches—linguistic patterns, crowdsourcing, and LLM prompts—for classifying software quality characteristics in user feedback, highlighting the effectiveness of crowdsourcing and LLMs.

Findings

Crowdsourcing achieved accuracy of 0.72 in classification.

LLMs matched crowdsourcing with accuracy around 0.66-0.68.

Linguistic pattern approach had limited potential with precision 0.38-0.92.

Abstract

Software qualities such as usability or reliability are among the strongest determinants of mobile app user satisfaction and constitute a significant portion of online user feedback on software products, making it a valuable source of quality-related feedback to guide the development process. The abundance of online user feedback warrants the automated identification of quality characteristics, but the online user feedback's heterogeneity and the lack of appropriate training corpora limit the applicability of supervised machine learning. We therefore investigate the viability of three approaches that could be effective in low-data settings: language patterns (LPs) based on quality-related keywords, instructions for crowdsourced micro-tasks, and large language model (LLM) prompts. We determined the feasibility of each approach and then compared their accuracy. For the complex multiclass classification of quality characteristics, the LP-based approach achieved a varied precision (0.38-0.92) depending on the quality characteristic, and low recall; crowdsourcing achieved the best average accuracy in two consecutive phases (0.63, 0.72), which could be matched by the best-performing LLM condition (0.66) and a prediction based on the LLMs' majority vote (0.68). Our findings show that in this low-data setting, the two approaches that use crowdsourcing or LLMs instead of involving experts achieve accurate classifications, while the LP-based approach has only limited potential. The promise of crowdsourcing and LLMs in this context might even extend to building training corpora.