Evaluating Software User Feedback Classifiers on Unseen Apps, Datasets, and Metadata

TL;DR

This study evaluates the generalization of machine learning classifiers for software user feedback across unseen apps, datasets, and metadata, revealing limited transferability and the potential of multi-dataset training.

Contribution

It provides an empirical assessment of classifier performance on unseen data and explores the impact of metadata and multi-dataset training on classification accuracy.

Findings

Classifiers perform similarly on unseen apps compared to seen apps.

Classifiers struggle with unseen datasets from different platforms or labels.

Multi-dataset training and zero-shot approaches can improve generalization.

Abstract

Listening to user's requirements is crucial to building and maintaining high quality software. Online software user feedback has been shown to contain large amounts of information useful to requirements engineering (RE). Previous studies have created machine learning classifiers for parsing this feedback for development insight. While these classifiers report generally good performance when evaluated on a test set, questions remain as to how well they extend to unseen data in various forms. This study evaluates machine learning classifiers performance on feedback for two common classification tasks (classifying bug reports and feature requests). Using seven datasets from prior research studies, we investigate the performance of classifiers when evaluated on feedback from different apps than those contained in the training set and when evaluated on completely different datasets (coming from different feedback platforms and/or labelled by different researchers). We also measure the difference in performance of using platform-specific metadata as a feature in classification. We demonstrate that classification performance is similar on feedback from unseen apps compared to seen apps in the majority of cases tested. However, the classifiers do not perform well on unseen datasets. We show that multi-dataset training or zero shot classification approaches can somewhat mitigate this performance decrease. Finally, we find that using metadata as features in classifying bug reports and feature requests does not lead to a statistically significant improvement in the majority of datasets tested. We discuss the implications of these results on developing user feedback classification models to analyse and extract software requirements.