Analysis of Problem Tokens to Rank Factors Impacting Quality in VoIP Applications

TL;DR

This paper investigates how problem tokens from user surveys can be used to identify and rank factors affecting call quality in VoIP applications, providing actionable insights for improving user experience.

Contribution

It introduces a method to rank and isolate independent call quality factors using problem tokens, validated with large-scale Skype survey data.

Findings

Problem tokens effectively identify key call quality issues.

The ranking method isolates independent impact factors.

Practical examples demonstrate actionable insights.

Abstract

User-perceived quality-of-experience (QoE) in internet telephony systems is commonly evaluated using subjective ratings computed as a Mean Opinion Score (MOS). In such systems, while user MOS can be tracked on an ongoing basis, it does not give insight into which factors of a call induced any perceived degradation in QoE -- it does not tell us what caused a user to have a sub-optimal experience. For effective planning of product improvements, we are interested in understanding the impact of each of these degrading factors, allowing the estimation of the return (i.e., the improvement in user QoE) for a given investment. To obtain such insights, we advocate the use of an end-of-call "problem token questionnaire" (PTQ) which probes the user about common call quality issues (e.g., distorted audio or frozen video) which they may have experienced. In this paper, we show the efficacy of this questionnaire using data gathered from over 700,000 end-of-call surveys gathered from Skype (a large commercial VoIP application). We present a method to rank call quality and reliability issues and address the challenge of isolating independent factors impacting the QoE. Finally, we present representative examples of how these problem tokens have proven to be useful in practice.