The Role of User Reviews in App Updates: A Preliminary Investigation on App Release Notes

TL;DR

This study investigates how user reviews influence app updates by analyzing Spotify's release notes and reviews, revealing that a significant portion of reviews are irrelevant, with relevant ones often posted before or after updates focusing on requests, bugs, or praise.

Contribution

It provides empirical insights into the relationship between user reviews and app updates, highlighting the limited relevance of many reviews to release notes.

Findings

Over 60% of matched reviews are irrelevant to release notes.

Relevant reviews are split evenly between pre- and post-update, focusing on requests and bug reports.

Many reviews posted before updates are about suggestions and complaints.

Abstract

Release planning for mobile apps has recently become an area of active research. Prior research in this area concentrated on the analysis of release notes and on tracking user reviews to support app evolution with issue trackers. However, little is known about the impact of user reviews on the evolution of mobile apps. Our work explores the role of user reviews in app updates based on release notes. For this purpose, we collected user reviews and release notes of Spotify, the 'number one' app in the 'Music' category in Apple App Store, as the research data. Then, we manually removed non-informative parts of each release note, and manually determined the relevance of the app reviews with respect to the release notes. We did this by using Word2Vec calculation techniques based on the top 80 app release notes with the highest similarities. Our empirical results show that more than 60% of the matched reviews are actually irrelevant to the corresponding release notes. When zooming in at these relevant user reviews, we found that around half of them were posted before the new release and referred to requests, suggestions, and complaints. Whereas, the other half of the relevant user reviews were posted after updating the apps and concentrated more on bug reports and praise.