Positivity Bias in Customer Satisfaction Ratings

TL;DR

This paper investigates the positivity bias in customer satisfaction ratings by analyzing chat logs with neural network models, revealing that unlabeled sessions likely had lower satisfaction than labeled, positively-rated sessions.

Contribution

It introduces RNN-based models to infer satisfaction levels of unlabeled customer sessions, uncovering biases in online rating data and providing insights for better customer experience management.

Findings

Labeled sessions show overwhelmingly positive ratings.

Most unlabeled sessions likely had lower satisfaction.

Data analytics can detect dissatisfied customers in real-time.

Abstract

Customer ratings are valuable sources to understand their satisfaction and are critical for designing better customer experiences and recommendations. The majority of customers, however, do not respond to rating surveys, which makes the result less representative. To understand overall satisfaction, this paper aims to investigate how likely customers without responses had satisfactory experiences compared to those respondents. To infer customer satisfaction of such unlabeled sessions, we propose models using recurrent neural networks (RNNs) that learn continuous representations of unstructured text conversation. By analyzing online chat logs of over 170,000 sessions from Samsung's customer service department, we make a novel finding that while labeled sessions contributed by a small fraction of customers received overwhelmingly positive reviews, the majority of unlabeled sessions would have received lower ratings by customers. The data analytics presented in this paper not only have practical implications for helping detect dissatisfied customers on live chat services but also make theoretical contributions on discovering the level of biases in online rating platforms.