Integrating Multi-Label Classification and Generative AI for Scalable Analysis of User Feedback

TL;DR

This paper combines multi-label classification and generative AI to analyze large volumes of user feedback efficiently, providing topic labels and summaries, and evaluates the relationship between sentiment and product satisfaction.

Contribution

It introduces a scalable approach integrating supervised multi-label classification and generative AI for user feedback analysis, enhancing interpretability and communication.

Findings

Generative AI effectively summarizes user comments.

Sentiment analysis alone is unreliable for measuring satisfaction.

Explicit surveys are necessary for accurate satisfaction assessment.

Abstract

In highly competitive software markets, user experience (UX) evaluation is crucial for ensuring software quality and fostering long-term product success. Such UX evaluations typically combine quantitative metrics from standardized questionnaires with qualitative feedback collected through open-ended questions. While open-ended feedback offers valuable insights for improvement and helps explain quantitative results, analyzing large volumes of user comments is challenging and time-consuming. In this paper, we present techniques developed during a long-term UX measurement project at a major software company to efficiently process and interpret extensive volumes of user comments. To provide a high-level overview of the collected comments, we employ a supervised machine learning approach that assigns meaningful, pre-defined topic labels to each comment. Additionally, we demonstrate how generative AI (GenAI) can be leveraged to create concise and informative summaries of user feedback, facilitating effective communication of findings to the organization and especially upper management. Finally, we investigate whether the sentiment expressed in user comments can serve as an indicator for overall product satisfaction. Our results show that sentiment analysis alone does not reliably reflect user satisfaction. Instead, product satisfaction needs to be assessed explicitly in surveys to measure the user's perception of the product.