A crowdsourcing approach to video quality assessment

TL;DR

This paper introduces an open-source, crowdsourcing-based extension of the ITU-T Rec. P.910 standard for video quality assessment, making subjective testing faster, cheaper, and more accessible.

Contribution

It presents a novel crowdsourcing approach that replicates traditional lab-based subjective video quality tests with validated accuracy and reproducibility.

Findings

Crowdsourcing extension matches lab-based results in accuracy.

The approach reduces cost and barriers to video quality assessment.

Validated high reproducibility across different conditions.

Abstract

The gold standard for measuring video quality is the subjective test, and the most prevalent is the ITU-T Rec. P.910, a lab-based subjective standard in use for the past two decades. However, in practice using ITU-T Rec. P.910 is slow, expensive, and requires a lab, which all create barriers to usage. As a result, most research papers that need to measure video quality don't use ITU-T Rec. P.910 but rather metrics that are not well correlated to subjective opinion. We provide an open-source extension of ITU-T Rec. P.910 based on crowdsourcing principles which address the speed, usage cost, and barrier to usage issues. We implement Absolute Category Rating (ACR), ACR with hidden reference (ACR-HR), Degradation Category Rating (DCR), and Comparison Category Rating (CCR). It includes rater, environment, hardware, and network qualifications, as well as gold and trapping questions to ensure quality. We have validated that the implementation is both accurate and highly reproducible compared to existing ITU-T Rec. P.910 lab studies.