Automating Explanation Need Management in App Reviews: A Case Study from the Navigation App Industry

TL;DR

This paper presents a semi-automated system for managing explanation needs in app reviews, classifying reviews with taxonomy and assigning them to relevant teams to improve response efficiency in the navigation app industry.

Contribution

It introduces a hierarchical assignment strategy leveraging taxonomy categories and external sources to automate review response management in a real-world setting.

Findings

Hierarchical assignment improved team identification accuracy to 79.2%.

External sources enhanced explanation comprehensiveness.

Challenges remain in interrater agreement and response generation.

Abstract

Providing explanations in response to user reviews is a time-consuming and repetitive task for companies, as many reviews present similar issues requiring nearly identical responses. To improve efficiency, this paper proposes a semi-automated approach to managing explanation needs in user reviews. The approach leverages taxonomy categories to classify reviews and assign them to relevant internal teams or sources for responses. 2,366 app reviews from the Google Play Store and Apple App Store were scraped and analyzed using a word and phrase filtering system to detect explanation needs. The detected needs were categorized and assigned to specific internal teams at the company Graphmasters GmbH, using a hierarchical assignment strategy that prioritizes the most relevant teams. Additionally, external sources, such as existing support articles and past review responses, were integrated to provide comprehensive explanations. The system was evaluated through interviews and surveys with the Graphmasters support team, which consists of four employees. The results showed that the hierarchical assignment method improved the accuracy of team assignments, with correct teams being identified in 79.2% of cases. However, challenges in interrater agreement and the need for new responses in certain cases, particularly for Apple App Store reviews, were noted. Future work will focus on refining the taxonomy and enhancing the automation process to reduce manual intervention further.