Objective video quality metrics application to video codecs comparisons: choosing the best for subjective quality estimation

TL;DR

This paper compares various versions of standard video quality metrics to identify the most relevant ones for codec comparison, using a dataset of encoded videos and subjective quality scores collected over several years.

Contribution

It provides a fundamental comparison of different metric versions to recommend the best for subjective video quality estimation in codec evaluations.

Findings

Identifies the most relevant metric versions for codec comparison

Analyzes the impact of different calculation rules on metric effectiveness

Recommends standardized metrics for consistent quality assessment

Abstract

Quality assessment plays a key role in creating and comparing video compression algorithms. Despite the development of a large number of new methods for assessing quality, generally accepted and well-known codecs comparisons mainly use the classical methods like PSNR, SSIM and new method VMAF. These methods can be calculated following different rules: they can use different frame-by-frame averaging techniques or different summation of color components. In this paper, a fundamental comparison of various versions of generally accepted metrics is carried out to find the most relevant and recommended versions of video quality metrics to be used in codecs comparisons. For comparison, we used a set of videos encoded with video codecs of different standards, and visual quality scores collected for the resulting set of streams since 2018 until 2021