Exploring Outliers in Crowdsourced Ranking for QoE

TL;DR

This paper introduces fast, iterative algorithms for outlier detection in crowdsourced QoE evaluation, achieving comparable robustness to existing methods but with significantly improved computational efficiency.

Contribution

It presents simple, nonconvex optimization-based algorithms for outlier detection that are faster and effective for robust QoE assessment in crowdsourcing data.

Findings

Algorithms achieve similar robustness to Huber-LASSO

Speed-up of 8 to 90 times over existing methods

Effective with or without prior knowledge of outlier sparsity

Abstract

Outlier detection is a crucial part of robust evaluation for crowdsourceable assessment of Quality of Experience (QoE) and has attracted much attention in recent years. In this paper, we propose some simple and fast algorithms for outlier detection and robust QoE evaluation based on the nonconvex optimization principle. Several iterative procedures are designed with or without knowing the number of outliers in samples. Theoretical analysis is given to show that such procedures can reach statistically good estimates under mild conditions. Finally, experimental results with simulated and real-world crowdsourcing datasets show that the proposed algorithms could produce similar performance to Huber-LASSO approach in robust ranking, yet with nearly 8 or 90 times speed-up, without or with a prior knowledge on the sparsity size of outliers, respectively. Therefore the proposed methodology provides us a set of helpful tools for robust QoE evaluation with crowdsourcing data.