BVI-UGC: A Video Quality Database for User-Generated Content Transcoding

TL;DR

This paper introduces BVI-UGC, a new video quality database for user-generated content transcoding, and evaluates the performance of various quality metrics, revealing their limitations in predicting perceptual quality.

Contribution

The creation of the BVI-UGC database with extensive subjective data and benchmarking of 21 quality metrics for UGC transcoding quality assessment.

Findings

Most quality metrics perform poorly with SROCC below 0.6

Full-reference and no-reference metrics both show limitations

The database enables better evaluation of quality assessment methods

Abstract

In recent years, user-generated content (UGC) has become one of the major video types consumed via streaming networks. Numerous research contributions have focused on assessing its visual quality through subjective tests and objective modeling. In most cases, objective assessments are based on a no-reference scenario, where the corresponding reference content is assumed not to be available. However, full-reference video quality assessment is also important for UGC in the delivery pipeline, particularly associated with the video transcoding process. In this context, we present a new UGC video quality database, BVI-UGC, for user-generated content transcoding, which contains 60 (non-pristine) reference videos and 1,080 test sequences. In this work, we simulated the creation of non-pristine reference sequences (with a wide range of compression distortions), typical of content uploaded to UGC platforms for transcoding. A comprehensive crowdsourced subjective study was then conducted involving more than 3,500 human participants. Based on this collected subjective data, we benchmarked the performance of 10 full-reference and 11 no-reference quality metrics. Our results demonstrate the poor performance (SROCC values are lower than 0.6) of these metrics in predicting the perceptual quality of UGC in two different scenarios (with or without a reference).