Recover Subjective Quality Scores from Noisy Measurements

TL;DR

This paper introduces a maximum likelihood estimation method to recover accurate subjective video quality scores from noisy measurements, improving confidence intervals and outlier handling compared to previous approaches.

Contribution

It presents a novel joint estimation framework that accounts for subject bias, consistency, and content ambiguity, providing closed-form confidence intervals and better data utilization.

Findings

Tighter confidence intervals for subjective quality estimates.

Improved outlier detection and handling of missing data.

Insights into subject bias and content variability.

Abstract

Simple quality metrics such as PSNR are known to not correlate well with subjective quality when tested across a wide spectrum of video content or quality regime. Recently, efforts have been made in designing objective quality metrics trained on subjective data (e.g. VMAF), demonstrating better correlation with video quality perceived by human. Clearly, the accuracy of such a metric heavily depends on the quality of the subjective data that it is trained on. In this paper, we propose a new approach to recover subjective quality scores from noisy raw measurements, using maximum likelihood estimation, by jointly estimating the subjective quality of impaired videos, the bias and consistency of test subjects, and the ambiguity of video contents all together. We also derive closed-from expression for the confidence interval of each estimate. Compared to previous methods which partially exploit the subjective information, our approach is able to exploit the information in full, yielding tighter confidence interval and better handling of outliers without the need for z-scoring or subject rejection. It also handles missing data more gracefully. Finally, as side information, it provides interesting insights on the test subjects and video contents.