KonVid-150k: A Dataset for No-Reference Video Quality Assessment of Videos in-the-Wild

TL;DR

This paper introduces KonVid-150k, a large and diverse in-the-wild video quality assessment dataset, and proposes new deep learning methods that significantly improve VQA performance and generalization across datasets.

Contribution

The creation of the KonVid-150k dataset and the development of MLSP-VQA approaches that outperform existing models in in-the-wild VQA tasks.

Findings

MLSP-VQA-FF achieves SRCC of 0.82 on KoNViD-1k.

MLSP-VQA-FF surpasses previous models in cross-dataset tests.

KonVid-150k enables better generalization for VQA models.

Abstract

Video quality assessment (VQA) methods focus on particular degradation types, usually artificially induced on a small set of reference videos. Hence, most traditional VQA methods under-perform in-the-wild. Deep learning approaches have had limited success due to the small size and diversity of existing VQA datasets, either artificial or authentically distorted. We introduce a new in-the-wild VQA dataset that is substantially larger and diverse: KonVid-150k. It consists of a coarsely annotated set of 153,841 videos having five quality ratings each, and 1,596 videos with a minimum of 89 ratings each. Additionally, we propose new efficient VQA approaches (MLSP-VQA) relying on multi-level spatially pooled deep-features (MLSP). They are exceptionally well suited for training at scale, compared to deep transfer learning approaches. Our best method, MLSP-VQA-FF, improves the Spearman rank-order correlation coefficient (SRCC) performance metric on the commonly used KoNViD-1k in-the-wild benchmark dataset to 0.82. It surpasses the best existing deep-learning model (0.80 SRCC) and hand-crafted feature-based method (0.78 SRCC). We further investigate how alternative approaches perform under different levels of label noise, and dataset size, showing that MLSP-VQA-FF is the overall best method for videos in-the-wild. Finally, we show that the MLSP-VQA models trained on KonVid-150k sets the new state-of-the-art for cross-test performance on KoNViD-1k, LIVE-VQC, and LIVE-Qualcomm with a 0.83, 0.75, and 0.64 SRCC, respectively. For both KoNViD-1k and LIVE-VQC this inter-dataset testing outperforms intra-dataset experiments, showing excellent generalization.