YouTube SFV+HDR Quality Dataset

TL;DR

This paper introduces the first large-scale dataset for evaluating the quality of short-form videos with HDR, analyzing subjective scores and the effectiveness of existing quality metrics.

Contribution

It presents a new SFV+HDR dataset with reliable subjective scores and a sampling framework, addressing the gap in quality assessment for this emerging video category.

Findings

SFV+HDR quality assessment differs from traditional UGC.

Existing UGC quality metrics are less effective for SFV+HDR.

The dataset enables better evaluation and development of quality metrics for SFV+HDR.

Abstract

The popularity of Short form videos (SFV) has grown dramatically in the past few years, and has become a phenomenal video category with billions of viewers. Meanwhile, High Dynamic Range (HDR) as an advanced feature also becomes more and more popular on video sharing platforms. As a hot topic with huge impact, SFV and HDR bring new questions to video quality research: 1) is SFV+HDR quality assessment significantly different from traditional User Generated Content (UGC) quality assessment? 2) do objective quality metrics designed for traditional UGC still work well for SFV+HDR? To answer the above questions, we created the first large scale SFV+HDR dataset with reliable subjective quality scores, covering 10 popular content categories. Further, we also introduce a general sampling framework to maximize the representativeness of the dataset. We provided a comprehensive analysis of subjective quality scores for Short form SDR and HDR videos, and discuss the reliability of state-of-the-art UGC quality metrics and potential improvements.