What can be learned from satisfaction assessments?

TL;DR

This paper investigates biases in customer satisfaction surveys, demonstrating how survey design and calibration can reduce errors and improve the extraction of meaningful insights from ordinal data.

Contribution

It reveals limitations of common segmentation practices and proposes calibration and binning strategies to enhance survey data analysis.

Findings

Uneven segmentation limits data utility

Calibration can assess irreducible error

Thoughtful design reduces non-systematic bias

Abstract

Companies survey their customers to measure their satisfaction levels with the company and its services. The received responses are crucial as they allow companies to assess their respective performances and find ways to make needed improvements. This study focuses on the non-systematic bias that arises when customers assign numerical values in ordinal surveys. Using real customer satisfaction survey data of a large retail bank, we show that the common practice of segmenting ordinal survey responses into uneven segments limit the value that can be extracted from the data. We then show that it is possible to assess the magnitude of the irreducible error under simple assumptions, even in real surveys, and place the achievable modeling goal in perspective. We finish the study by suggesting that a thoughtful survey design, which uses either a careful binning strategy or proper calibration, can reduce the compounding non-systematic error even in elaborated ordinal surveys. A possible application of the calibration method we propose is efficiently conducting targeted surveys using active learning.