Mining User Privacy Concern Topics from App Reviews

TL;DR

This paper presents an automated method combining information retrieval, supervised classification, and interpretable topic modeling to effectively identify and analyze user privacy concerns in app reviews, outperforming existing baselines.

Contribution

The study introduces a novel integrated approach for mining privacy concern topics from app reviews, including a new topic detection algorithm with superior coherence and diversity.

Findings

Achieves 96.80% precision in retrieving privacy reviews

Supervised classifiers reach over 91% F1 score in privacy review detection

Proposed topic mining outperforms LDA in coherence and diversity

Abstract

Context: As mobile applications (Apps) widely spread over our society and life, various personal information is constantly demanded by Apps in exchange for more intelligent and customized functionality. An increasing number of users are voicing their privacy concerns through app reviews on App stores. Objective: The main challenge of effectively mining privacy concerns from user reviews lies in the fact that reviews expressing privacy concerns are overridden by a large number of reviews expressing more generic themes and noisy content. In this work, we propose a novel automated approach to overcome that challenge. Method: Our approach first employs information retrieval and document embeddings to unsupervisedly extract candidate privacy reviews that are further labeled to prepare the annotation dataset. Then, supervised classifiers are trained to automatically identify privacy reviews. Finally, we design an interpretable topic mining algorithm to detect privacy concern topics contained in the privacy reviews. Results: Experimental results show that the best performed document embedding achieves an average precision of 96.80% in the top 100 retrieved candidate privacy reviews. All of the trained privacy review classifiers can achieve an F1 value of more than 91%, outperforming the recent keywords matching baseline with the maximum F1 margin being 7.5%. For detecting privacy concern topics from privacy reviews, our proposed algorithm achieves both better topic coherence and diversity than three strong topic modeling baselines including LDA. Conclusion: Empirical evaluation results demonstrate the effectiveness of our approach in identifying privacy reviews and detecting user privacy concerns expressed in App reviews.