The Impact of Annotation Guidelines and Annotated Data on Extracting App Features from App Reviews

TL;DR

This paper investigates how annotation guidelines and data sources affect the quality of app feature extraction from reviews, proposing guideline improvements and analyzing training data requirements for better model performance.

Contribution

It introduces revised annotation guidelines that produce more informative features and evaluates the impact of different training data sources, including cross-category reviews and product reviews.

Findings

Simulated guideline changes yield less noisy, more useful features.

Training on reviews from the test app improves recall.

Augmenting training data with product reviews increases recall but reduces precision.

Abstract

Annotation guidelines used to guide the annotation of training and evaluation datasets can have a considerable impact on the quality of machine learning models. In this study, we explore the effects of annotation guidelines on the quality of app feature extraction models. As a main result, we propose several changes to the existing annotation guidelines with a goal of making the extracted app features more useful and informative to the app developers. We test the proposed changes via simulating the application of the new annotation guidelines and then evaluating the performance of the supervised machine learning models trained on datasets annotated with initial and simulated guidelines. While the overall performance of automatic app feature extraction remains the same as compared to the model trained on the dataset with initial annotations, the features extracted by the model trained on the dataset with simulated new annotations are less noisy and more informative to the app developers. Secondly, we are interested in what kind of annotated training data is necessary for training an automatic app feature extraction model. In particular, we explore whether the training set should contain annotated app reviews from those apps/app categories on which the model is subsequently planned to be applied, or is it sufficient to have annotated app reviews from any app available for training, even when these apps are from very different categories compared to the test app. Our experiments show that having annotated training reviews from the test app is not necessary although including them into training set helps to improve recall. Furthermore, we test whether augmenting the training set with annotated product reviews helps to improve the performance of app feature extraction. We find that the models trained on augmented training set lead to improved recall but at the cost of the drop in precision.