Beyond Correlation: Evaluating Multimedia Quality Models with the Constrained Concordance Index

TL;DR

This paper introduces the Constrained Concordance Index (CCI), a new metric for evaluating multimedia quality models that accounts for subjective rating uncertainties and statistical significance, improving evaluation accuracy.

Contribution

The paper proposes the CCI metric, addressing limitations of traditional correlation measures by incorporating significance testing and confidence interval considerations.

Findings

CCI outperforms traditional metrics in low sample scenarios

Incorporating rater variability improves evaluation robustness

Focusing on significant MOS differences enhances model assessment

Abstract

This study investigates the evaluation of multimedia quality models, focusing on the inherent uncertainties in subjective Mean Opinion Score (MOS) ratings due to factors like rater inconsistency and bias. Traditional statistical measures such as Pearson's Correlation Coefficient (PCC), Spearman's Rank Correlation Coefficient (SRCC), and Kendall's Tau (KTAU) often fail to account for these uncertainties, leading to inaccuracies in model performance assessment. We introduce the Constrained Concordance Index (CCI), a novel metric designed to overcome the limitations of existing metrics by considering the statistical significance of MOS differences and excluding comparisons where MOS confidence intervals overlap. Through comprehensive experiments across various domains including speech and image quality assessment, we demonstrate that CCI provides a more robust and accurate evaluation of instrumental quality models, especially in scenarios of low sample sizes, rater group variability, and restriction of range. Our findings suggest that incorporating rater subjectivity and focusing on statistically significant pairs can significantly enhance the evaluation framework for multimedia quality prediction models. This work not only sheds light on the overlooked aspects of subjective rating uncertainties but also proposes a methodological advancement for more reliable and accurate quality model evaluation.