Beyond Keywords: A Context-based Hybrid Approach to Mining Ethical Concern-related App Reviews

TL;DR

This paper introduces a hybrid NLP approach combining NLI and LLMs to effectively extract ethical concern-related app reviews, especially privacy issues, from large datasets, surpassing keyword-based methods.

Contribution

It presents a novel combination of NLI and LLM techniques for scalable extraction of ethical concerns in app reviews, with domain-specific evaluation and improved results.

Findings

DeBERTa-v3-base-mnli-fever-anli outperformed other NLI models.

Llama3.1-8B-Instruct was the best LLM for review classification.

Additional 1,008 privacy reviews were identified beyond keyword methods.

Abstract

With the increasing proliferation of mobile applications in our everyday experiences, the concerns surrounding ethics have surged significantly. Users generally communicate their feedback, report issues, and suggest new functionalities in application (app) reviews, frequently emphasizing safety, privacy, and accountability concerns. Incorporating these reviews is essential to developing successful products. However, app reviews related to ethical concerns generally use domain-specific language and are expressed using a more varied vocabulary. Thus making automated ethical concern-related app review extraction a challenging and time-consuming effort. This study proposes a novel Natural Language Processing (NLP) based approach that combines Natural Language Inference (NLI), which provides a deep comprehension of language nuances, and a decoder-only (LLaMA-like) Large Language Model (LLM) to extract ethical concern-related app reviews at scale. Utilizing 43,647 app reviews from the mental health domain, the proposed methodology 1) Evaluates four NLI models to extract potential privacy reviews and compares the results of domain-specific privacy hypotheses with generic privacy hypotheses; 2) Evaluates four LLMs for classifying app reviews to privacy concerns; and 3) Uses the best NLI and LLM models further to extract new privacy reviews from the dataset. Results show that the DeBERTa-v3-base-mnli-fever-anli NLI model with domain-specific hypotheses yields the best performance, and Llama3.1-8B-Instruct LLM performs best in the classification of app reviews. Then, using NLI+LLM, an additional 1,008 new privacy-related reviews were extracted that were not identified through the keyword-based approach in previous research, thus demonstrating the effectiveness of the proposed approach.