On Design of Problem Token Questions in Quality of Experience Surveys

TL;DR

This paper investigates how question order bias affects QoE surveys and proposes a method to select a small, highly informative subset of problem tokens using greedy maximization, based on extensive live system data.

Contribution

It demonstrates that randomizing question order reduces bias without affecting response levels and introduces an effective greedy method for selecting impactful survey questions.

Findings

Randomizing token order reduces bias significantly.

Users respond similarly to fixed and randomized question orders.

Selecting 30% of questions captures 94% of information.

Abstract

User surveys for Quality of Experience (QoE) are a critical source of information. In addition to the common "star rating" used to estimate Mean Opinion Score (MOS), more detailed survey questions (problem tokens) about specific areas provide valuable insight into the factors impacting QoE. This paper explores two aspects of the problem token questionnaire design. First, we study the bias introduced by fixed question order, and second, we study the challenge of selecting a subset of questions to keep the token set small. Based on 900,000 calls gathered using a randomized controlled experiment from a live system, we find that the order bias can be significantly reduced by randomizing the display order of tokens. The difference in response rate varies based on token position and display design. It is worth noting that the users respond to the randomized-order variant at levels that are comparable to the fixed-order variant. The effective selection of a subset of token questions is achieved by extracting tokens that provide the highest information gain over user ratings. This selection is known to be in the class of NP-hard problems. We apply a well-known greedy submodular maximization method on our dataset to capture 94% of the information using just 30% of the questions.