On the Emotion of Users in App Reviews

TL;DR

This paper analyzes over seven million Apple App Store reviews to explore the emotional sentiment expressed by users, aiming to determine its usefulness for software engineering insights and identifying potential pitfalls.

Contribution

It provides an empirical analysis of user emotions in app reviews and evaluates their potential as informative features for software engineering.

Findings

Emotional sentiment varies significantly across reviews.

Sentiment analysis can offer valuable insights but has limitations.

Initial results highlight both opportunities and challenges in using sentiment data.

Abstract

App store analysis has become an important discipline in recent software engineering research. It empirically studies apps using information mined from their distribution platforms. Information provided by users, such as app reviews, are of high interest to developers. Commercial providers such as App Annie analyzing this information became an important source for companies developing and marketing mobile apps. In this paper, we perform an exploratory study, which analyzes over seven million reviews from the Apple AppStore regarding their emotional sentiment. Since recent research in this field used sentiments to detail and refine their results, we aim to gain deeper insights into the nature of sentiments in user reviews. In this study we try to evaluate whether or not the emotional sentiment can be an informative feature for software engineers, as well as pitfalls of its usage. We present our initial results and discuss how they can be interpreted from the software engineering perspective.

Tables

Table 1. TABLE I: User Reviews by Categories. N = 7,396,551. ■ ■ \blacksquare = Positive, ■ ■ \blacksquare = Neutral, ■ ■ \blacksquare = Negative, ■ ■ \blacksquare = Undefined.

#	App Category	# Reviews Free Apps	# Reviews Paid Apps	# Reviews Sentiment	Mean	SD	Median	Distribution
1	Books	111,111	6,686	116,947	1.628	2.437	3.0
2	Business	15,539	23,514	38,984	1.878	2.125	3.0
3	Catalogs	5,586	328	5,908	2.137	1.726	3.0
4	Education	86,858	9,046	95,684	2.449	1.681	3.0
5	Entertainment	116,613	16,812	132,845	0.619	2.663	2.0
6	Finance	190,055	9,303	199,054	1.796	2.126	3.0
7	Food & Drink	297,732	3,784	300,848	2.203	1.893	3.0
8	Games	189,102	320,296	506,947	1.515	2.288	2.0
9	Health & Fitness	357,377	16,003	372,441	2.359	1.769	3.0
10	Lifestyle	196,733	4,807	201,029	1.683	2.252	3.0
11	Medical	17,054	4,530	21,508	1.932	2.214	3.0
12	Music	792,204	26,209	815,931	2.150	2.041	3.0
13	Navigation	293,854	8,562	301,625	2.044	1.947	3.0
14	News	352,659	20,565	372,258	1.121	2.488	2.0
15	Photo & Video	601,687	45,637	642,951	0.429	2.829	0.0
16	Productivity	127,440	9,796	136,828	1.382	2.368	2.0
17	Reference	505,966	20,145	525,456	2.736	1.334	3.0
18	Shopping	129,449	2,192	131,227	2.022	2.181	3.0
19	Soc. Networking	1,763,399	1,970	1,759,810	0.993	2.634	2.0
20	Sports	174,046	3,483	177,055	0.959	2.617	2.0
21	Travel	139,285	3,373	142,278	1.442	2.360	3.0
22	Utilities	106,584	9,522	115,710	1.305	2.426	2.0
23	Weather	237,739	21,916	258,377	1.273	2.463	2.0
		$\sum 6,808,072$	$\sum 588,479$	$\sum 7,371,701$	$\varnothing 1.845$	$\varnothing 2.211$	$\varnothing 2.522$

Table 2. TABLE II: Dispersion of Sentiments.

Topic	# Reviews	Range	IQR (Q.75 - Q.25)	SD
Bug Report	379	10	5	2.62
Feature Request	295	9	4	2.35
User Experience	735	10	1	1.99
Rating	2,604	10	1	1.90

Full text

I Introduction

In the past decade, app stores like Google Play or Apple AppStore became an important source for software engineers. Initially intended as software distribution platforms, their social aspects, such as the possibility for users to rate and review apps gained attention in the community. User reviews include useful information such as bug reports, feature requests or user experience [18]. These reviews have been evaluated in different ways, including general exploratory studies [18], classification [15], feature extraction [7], review filtering [3], and summarization [4]. However, user reviews are not mundane technical descriptions but also contain emotion. Many of the studies additionally applied sentiment analysis tools or APIs to improve their results. It remains unclear if existing tools are suitable for software engineering purposes [11, 12]. For this reason, we present this exploratory study, and focus on the sentiment, to find out if positive or negative emotions in user reviews can be used as an additional feature in app store analysis. We intend to discover links between informational value and the emotion of user reviews. A special focus is placed on monitoring the evolution of the sentiment over time, to discover patterns and to draw conclusions by manually analyzing these patterns.

The remainder of this paper is organized as follows. Section II presents related work in which emotion and sentiment has been used within the field of software engineering. In Section III we describe the design of the study, the collection and preparation of data, as well as the data analysis techniques used. In Section IV we provide a brief descriptive overview of the data set, present initial findings, and draw conclusions from the software engineering perspective. We conclude with a discussion and directions for future work in Section V.

II Related Work

Natural language processing has been used for sentiment analysis for more than a decade [21]. With the rise of social media it gained increasing importance. Initially intended for the use in marketing or political opinion mining, the topic became popular in many other social domains. This also applied for social aspects of software engineering as the community recognized its potential. Most of the research has been conducted in the field of developers’ interactions within bug trackers, commit messages or software related Q&A sites [17, 6]. With the growing interest in app store mining [3, 8, 10], emotional sentiment has been used in research related to user reviews in app stores. In this area, sentiment analysis mostly played a supportive role to detail findings or to increase the quality of results. Goul et al. used sentiment analysis on user reviews to address current bottlenecks in requirements[5]. Li et al. calculated the overall user satisfaction in the app stores [13]. A more fine-grained study by Guzman and Maalej used sentiment analysis to summarize the opinion on app features retrieved from user reviews [7]. Maalej et al. added sentiments as an informative feature to train a classifier used to categorize user reviews regarding different topics [15, 14]. However, Jongeling et al. reported on possible negative results when using sentiment analysis tools for software engineering research [12]. The use of sentiment analysis tools and APIs is highly dependent on the use case, and those have a significant impact on study results [11]. Fu et al. [4] built a regression model of words based on common user vocabulary and identified words with outstanding positive and negative sentiment based on their sample. Our study differentiates from the earlier studies therein that we undertake an isolated look on sentiments of user reviews in order to better understand if and how the sentiment can be used as a supportive feature for app store analyses.

III Study Design

The goal of this exploratory study is to find out whether and when considering the emotionality of app reviews can help to gain a deeper understanding of users’ needs from the software engineering perspective. This section describes our research method and data set. Our research method consists of a data collection phase and a data analysis phase, depicted in Figure 1. To evaluate the emotion within user reviews we measure the strength of the sentiment using SentiStrength 2.2 with its latest configuration files from September 21, 2011. SentiStrength is a lexical sentiment extraction tool with human-level accuracy that can be applied to short informal texts written in English [20]. It was trained for short social web texts such as Twitter. SentiStrength has the highest average accuracy among 15 Twitter sentiment analysis tools [1]. Since user reviews in app stores have approximately an average length of a tweet (average length in our data set is 165) and an informal style, we chose SentiStregth over comparable tools. Alternative tools might generate other results in detail, but we assume the overall results will correlate.

III-A Data Collection

For our study, we selected the top five free and paid apps (by December 18, 2016) for each of the 25 categories of the Apple AppStore. We chose the storefront of the United States to obtain reviews in English. For each app we gathered its details using the iTunes Search API111https://affiliate.itunes.apple.com/resources/documentation/itunes-store-web-service-search-api. An app detail consists of 44 values, such as the name, version, description, category, or price. The app reviews were programmatically scraped using a self developed tool, which accesses an internal iTunes API. A review consists of 10 values, including the reviewer name, title, description, rating, or date. The update dates of apps were manually extracted from the AppStore for selected apps, as they are not available over the iTunes API. These indicate when a specific app version has been released. In addition, we collected the release notes for each version describing the introduced changes. For further analysis the data was persisted inside a MongoDB database. To enable replication, our data set is publicly available on the project website222https://mast.informatik.uni-hamburg.de/app-review-analysis.

III-B Data Analysis

III-B1 Process Emojis

Emojis, a new form of emoticons, are increasingly being used by reviewers. These are unicode graphic symbols, of which 1,851 different characters exist. In contrast to emoticons their emotional content, due to their huge variety, in many cases remains unclear. To our knowledge, current sentiment analysis tools do not consider emojis. Therefore we extended SentiStrength to support emojis. Novak et al. let 83 human annotators label over 1.6 million tweets in 13 European languages by the semtiment polarity (negative, neutral, or positive) [16]. While there exists no significant difference between the languages, there is a clear difference between tweets without emojis and tweets containing emojis. The authors provide a emoji sentiment lexicon including 751 emojis giving each of them a score of either -1, 0, or 1. From the list we selected all emojis with an occurrence above 100 in the labeled tweets, resulting in 214 emojis. Using the list we replaced each of the 214 emojis with an emoticon (text representation) in order for SentiStregth to be able to consider the emojis while calculating the sentiment.

III-B2 Calculate Sentiment

We provide each review (combination of title and description) as input to SentiStrength. Regarding mixed emotions [2], SentiStrength returns the sentiment as two scores: a negative score from -1 (not negative) to -5 (extremely negative) and a positive score from 1 (not positive) to 5 (extremely positive). The review "I hate that u need wifi but it is great.", e.g., has a positive strength of 3 and a negative strength of -4. The sentiment is calculated by matching each token within an input to a set of dictionaries, included in the configuration files. Each dictionary defines fixed scores for specific tokens: "I hate[-4] that u need wifi but it is great[+3].". The overall score of an input is determined by using the maximum and minimum score of all annotated tokens, in this case [+3, -4]. The dictionaries also include scores for emoticons, booster words which can increase or decrease a token score ("extremely good[3] [+2 booster word]"), or lookup tables for slang words.

III-B3 Calculate Combined Sentiment

Finally, we computed a combined score for each review using an approach derived from Thelwall et al. [19]. Considering both the negative score (n) and the positive score (p) we chose p as the combined sentiment if p + n > 0. If p + n < 0 we set n as the combined sentiment. In case p = -n and p < 4 we set the combined sentiment to 0 and assume the review to be neutral. If p = -n and p >= 4 we set the combined sentiment to undefined and the reviews are removed from the data set. In contrary to [19] we use a scale from -5 to 5 instead of a scale from -1 to 1. Our example review with scores [+3, -4] received a combined score of -4.

IV Results

IV-A Research Data

An overview of our data set is depicted in Table I. We collected 7,396,551 reviews from 245 applications (125 free, 120 paid) in total. The data set consists of 23 distinct categories. The categories "Kids" and "Magazines & Newspapers" are not present in our data set, as their top five free/paid apps were of another primary category, such as "News", "Entertainment", or "Education". Another five paid apps appeared in the top five apps of two categories, resulting in 120 paid apps in total. For each app we collected all reviews beginning with the apps’ first release. The oldest feedback was provided in July, 2008. Overall, our data set spans more than 8 years.

In our data set most reviews for free apps were submitted in the category "Social Networks" (1,763,399 - 25.90%), least in the category "Catalogs" (5,586 - 0.08%). Most paid apps reviews were written in the category "Games" (320296 - 54.43%), least in the category "Catalogs" (328 - 0.06%). We could calculate the sentiment for 7,371,701 apps, only 24.850 reviews could not be classified (0.003%)

The average length of a review (combined of title and description) is 165 characters. The average rating is 3.819 stars. The average sentiment scores are [+2.535, -1.562]. The combined sentiment average is 1.544. Both sentiments clearly highlight a more positive than negative emotion within app reviews. Only 214 reviews (0.003%) received a vote of another app user, indicating that the provided review is helpful.

IV-B Emotion and the Rating

Besides the review, a user can give a star rating on a scale from 1 to 5 stars. We assume that a higher star rating correlates with stronger positive emotions and vice versa, that a low star rating correlates with a negative emotion. Figure 2 shows a box plot of the sentiment for each star rating. At a first glance the plot seems to confirm our assumption. The Pearson correlation coefficient of 0.5699 also draws attention towards a weak positive linear correlation. Although this finding is not surprising, since it follows the intuitional assumption, we will take a closer look at the outliers. Even though the Spearman rank correlation is influenced less by outliers, it also draws a weak correlation (0.560776). Especially for the four and five star ratings, many reviews are classified as outliers. Out of 612,271 reviews rated with five stars, 199,194 have a combined score of -4 or -5 (32.53%). This can be explained by the nature of SentiStrength: "I would be very sad without it" has a rating of five stars, but the sentiscore has positive strength 1 and negative strength -5, as a consequence of the booster word "very" and the negative word "sad".

This shows that on average sentiment and the star rating are correlated, but are not a sound measure to determine if a user is happy with the app. This affects studies that rely on the link between emotion and user satisfaction.

IV-C Emotion and the Price

We raise the question if users react emotionally stronger when money is involved. Does a user react with more emotion to a bug or a new added feature when the app was subject to a charge, and does the price make a difference? Harman et al. [9] showed that there is no correlation between price and downloads, nor between price and rating. Pagano and Maalej [18] found that there is significant increase in feedback length between lower-price and higher-price applications, which we can confirm based on our sample. We can reject the hypothesis that there is a significant linear correlation between price and emotion (Pearson/Spearman: 0.021/0.026). A possible explanation might be that users, who are generally willing to provide feedback do not differentiate between free, low-price, and higher-priced apps and give feedback based on an intrinsic motivation. Their inclination of expression stays the same, independent of the pricing.

IV-D Emotion and the Content

Pagano and Maalej [18] described 17 different topics in user reviews. Especially for software engineers it is important to filter by the topics Bug Reports, Feature Requests and User Experience. Maalej and Nabil [15] compared the accuracy of different techniques for the classification into the topics: Bug Reports, Feature Requests, User Experience and Rating. A rating here is a literal repetition of the star rating including praise, dispraise, a distractive critique, or a dissuasion. The authors used bag of words with and without lemmatization and stopword removal, the star rating, tense and the sentiment. They trained a classifier with different combinations of these features. The authors found that there is not a one-size-fits-all solution. The sentiment could be used to achieve a higher accuracy in some cases. At this, it did not make a difference if they used the two sided sentiment or the combined sentiment. However, not in every case could the use of the sentiment as an additional feature could lead to better results. Maalej et al. [14] showed that other features are more informative for the classification. Only for the topics User Experience and Bug Report, the sentiment was ranked amongst the 10 most important features. We analyzed the sentiment based on their labeled dataset and found that on average a bug report has a more negative sentiment than a user experience (cp. Figure 3). A closer look on the dispersion measures reveals that the sentiments are very scattered, especially for the topics Bug Report and Feature Request (cp. Table II). Ratings and User Experience have a very high amount of outliers.

We think that the sentiment can be a much more important informative feature when the SentiStrenght tool is adjusted. Since SentiStrength was not intended for software engineering purposes, words that have a relative negative sentiment in the domain of software engineering are not necessarily negative by their nature. For example, the fictitious user review "This app is really buggy and crashes all the time on my phone" would receive a neutral sentiment [-1, 1]. In the software engineering domain, words like bug or crash have a very negative connotation.

IV-E Patterns in Emotions

The emotionality of app reviews can vary over time. Reasons for this can be manifold, e.g., features are added, bugs occur, or feature requests are ignored by app developers.

To understand the development of emotion, we selected all apps of our data set having more than 1,000 reviews within the timeframe from January 4, 2016 to December 18, 2016. For each app, we plotted the sentiment for the given timeframe. In these plots we found six reoccurring patterns, depicted in Figure 4. First, Consistent Emotion with a stable sentiment. Second, Inconsistent Emotion, which is less constant but does not show a clear positive or negative trend. Third, Sentiment Drops that show a sudden decrease. Fourth, Sentiment Jumps where the sentiment shows a sudden positive increase. The fifth and sixth pattern, Steady Decrease/Increase, show a constant negative or positive trend. Whereas the first two patterns do not have additional value for the developers, the observation of the latter can give additional insights. Particularly, when emotional drops or jumps occur after app updates, developers quickly receive feedback regarding the introduced changes.

We manually classified 101 plots of the evolution of emotion over one year (grouped by weekly average) into the aforementioned classes. When multiple patterns during the time period occured, we classified them into multiple classes. Most of the plots (62%) showed a inconsistent emotional pattern. 15% had a significant jump in emotion and 9% showed a drop, while 10% had a steady decrease, and only 3% a steady increase. 15% contained a consistent high or low emotion.

We took a detailed look into the reviews of two apps within the categories emotion jump and emotion drop to better understand the emergence of such patterns. We assume these categories to be the most interesting for developers. Figure 5 shows the evolution of the sentiment for the "Bank of America" app, while Figure 6 shows the sentiment for the "Gmail" app. In both figures we highlighted the release of app updates using grey vertical lines.

IV-E1 Bank of America

For this app 6,307 reviews were submitted in 2016. Figure 5 can be separated into five sections. Section 1 ranges from week 1 to week 12 with an inconsistent sentiment oscillating around 0. App updates including minor improvements released in week 3, 5 and 12 temporarily raise the sentiment above 0, afterwards the sentiment becomes negative again. The reviewers have extremely contrary positions, while some write "Good App!" others report "I rather run 10 miles to a branch, barefoot on snow, than use this app.". Section 2 (week 13 - 16) is initiated by a new app update and has a sentiment below 0. Mostly iPhone 6s users report that the app crashes. In the beginning of section 3 (week 17 - 31) two app updates are released, which introduce the support for iOS 9 (preinstalled on iPhone 6s). As a result, the sentiment increases with the most positive reviews in week 25, overall the sentiment in this section is positive in every week. Section 4 (week 32 - 34) has a negative sentiment. It is initiated by two app updates that introduce a redesign of the app and a minimum required iOS version of 8. Users mostly state "iOS 8 now required! […]", as a consequence they report "Doesn’t work with iPhone 4 […]". In the last section (week 35 - 50) the sentiment is above 0 again. In week 38 an app update is released, which introduces new features and causes the sentiment to further increase. Overall the reviews become shorter and less informative, mostly depicting the subjective impression of the users such as "Solid app", "Love it!!!", or "Great app".

IV-E2 Gmail

In 2016 users submitted 11,629 reviews for the app. Figure 6 can be divided into two sections. The first section ranges from week 1 to week 44 with an ongoing positive sentiment. The second section ranges from week 45 to week 50 with an enduring negative sentiment. In section 1 users mostly submit positive reviews, such as "This app does it all!", with an average length of 145 characters. In this section there is only one remarkably low sentiment in week 5 where users report issues when loading and deleting emails. We expect these issues to be caused by server failures, as there were no app updates released. Section 2 begins with the release of a redesigned app update. Overall, users submit 6,104 reviews in this section (1,017 reviews/week on average) compared to section 1 with 5,525 reviews (126 reviews/week on average). The average length of the reviews also increases to 273 characters, nearly twice as long as in section 1. Until week 47 the sentiment becomes more negative, users mostly complain about features that have been changed "When Deleting messages now requires an extra step - you failed!", or features that have been removed such as "Bring back Mark as Unread. Why on earth would you remove this feature?" and "The new app no longer allows me to select multiple emails.". In week 48 the app developers react with weekly updates integrating features as requested by the users, e.g., version 5.0.6 (November 30, 2016): "Select multiple messages […]", and "Mark as read/unread: […]". Starting with this release the sentiment becomes less negative.

Summarizing our results, it can be said that app updates have the most influence on the emotion in app reviews. Integrating single features as requested by users positively influences the sentiment, while, e.g., releasing a completely redesigned app can cause the previously positive sentiment to turn into a negative sentiment. This also applies for changing apps’ requirements such as the minimum supported iOS version. When providing negative reviews users write longer and more detailed reviews. After releasing major updates the number of reviews per week also increases for a specific period of time.

V Discussion and Future Directions

We found that emotion in user reviews can be seen as a meaningful additional meta-data. The fact that we could only observe weak correlations with other indicators tempers the expectations of current approaches. This can be linked back to the general approach of tools like SentiStrength. In future work it will be necessary to adjust tools and approaches to better fit the contents in user reviews. The correlation between the emotional sentiment and the user ratings is only weakly defined. Especially for very positive ratings with four and five stars, the rating itself is a better indicator for the users’ satisfaction than the sentiment. The emotional sentiment is not influenced by the price. While reviews for higher priced apps tend to be longer on average, users do not react with more emotion, neither in a positive nor in a negative way. We assume that the price paid for an app does not give additional incentives to review apps and that users who are willing to provide feedback have other motivations, which are not based on the price. The sentiment becomes more relevant when user reviews are classified based on their topic. We see a potential to further increase the quality of classification techniques when tools, such as SentiStrength, are adjusted to software topics by applying different sentiment weights to given buzzwords such as "bug" or "crash". A promising insight is the development of the sentiment over time and the reoccurring patterns that could be found in our analysis. Particularly mixed emotions attracted our interest, since the manual analysis indicates that users actively discuss pros and cons of a release. We assume that adding additional features to the timeline, we can give more exact assumptions about the users perception of a release. Our initial findings motivate to adjust sentiment analysis towards a better fit for the purpose of software reviews. In this study we used SentiStrength knowingly without further modification. We assume that especially the emotional weighting of terms must be adjusted.

In this initial study we found that adding the emotion of a review as a feature can provide additional information. Whereas some of our findings confirmed intuitive assumptions, others revealed that the calculated sentiment often diverges.