Fast Adaptive Algorithm for Robust Evaluation of Quality of Experience

TL;DR

This paper introduces iLTS, an efficient iterative algorithm for outlier detection and robust Quality of Experience evaluation, capable of automatic outlier detection and significantly faster than existing methods.

Contribution

The paper presents a novel iterative Least Trimmed Squares algorithm that improves robustness and speed in outlier detection for crowdsourceable QoE evaluation.

Findings

iLTS is up to 190 times faster than LASSO-based methods.

The method effectively detects outliers without prior knowledge of their quantity.

Demonstrated success on simulated and real-world data.

Abstract

Outlier detection is an integral part of robust evaluation for crowdsourceable Quality of Experience (QoE) and has attracted much attention in recent years. In QoE for multimedia, outliers happen because of different test conditions, human errors, abnormal variations in context, {etc}. In this paper, we propose a simple yet effective algorithm for outlier detection and robust QoE evaluation named iterative Least Trimmed Squares (iLTS). The algorithm assigns binary weights to samples, i.e., 0 or 1 indicating if a sample is an outlier, then the outlier-trimmed subset least squares solutions give robust ranking scores. An iterative optimization is carried alternatively between updating weights and ranking scores which converges to a local optimizer in finite steps. In our test setting, iLTS is up to 190 times faster than LASSO-based methods with a comparable performance. Moreover, a varied version of this method shows adaptation in outlier detection, which provides an automatic detection to determine whether a data sample is an outlier without \emph{a priori} knowledge about the amount of the outliers. The effectiveness and efficiency of iLTS are demonstrated on both simulated examples and real-world applications. A Matlab package is provided to researchers exploiting crowdsourcing paired comparison data for robust ranking.