Hacking VMAF and VMAF NEG: vulnerability to different preprocessing methods

TL;DR

This paper reveals vulnerabilities in VMAF and VMAF NEG metrics, demonstrating how preprocessing can artificially inflate scores without improving actual visual quality, thus impacting video quality assessment reliability.

Contribution

The study uncovers how specific preprocessing techniques can significantly manipulate VMAF and VMAF NEG scores, highlighting potential flaws in current video quality measurement methods.

Findings

Preprocessing can increase VMAF scores by up to 218.8%.

Most preprocessing methods do not improve visual quality.

VMAF NEG scores can be artificially increased by up to 23.6%.

Abstract

Video-quality measurement plays a critical role in the development of video-processing applications. In this paper, we show how video preprocessing can artificially increase the popular quality metric VMAF and its tuning-resistant version, VMAF NEG. We propose a pipeline that tunes processing-algorithm parameters to increase VMAF by up to 218.8%. A subjective comparison revealed that for most preprocessing methods, a video's visual quality drops or stays unchanged. We also show that some preprocessing methods can increase VMAF NEG scores by up to 23.6%.