On Optimizing Image Codecs for VMAF NEG: Analysis, Issues, and a Robust Loss Proposal

TL;DR

This paper analyzes the vulnerabilities of VMAF NEG in image codec optimization, proposes a robust loss function incorporating VMAF NEG to improve fine-tuning, and validates the approach with both quantitative and perceptual results.

Contribution

It identifies vulnerabilities in VMAF NEG, introduces a robust loss for codec fine-tuning, and demonstrates improved perceptual quality through experiments.

Findings

VMAF NEG remains vulnerable to specific attacks.

The proposed robust loss enhances fine-tuning stability.

Perceptual evaluations show improved image quality.

Abstract

The VMAF (video multi-method assessment fusion) metric for image and video coding recently gained more and more popularity as it is supposed to have a high correlation with human perception. This makes training and particularly fine-tuning of machine-learned codecs on this metric interesting. However, VMAF is shown to be attackable in a way that, e.g., unsharpening an image can lead to a gain in VMAF quality while decreasing the quality in human perception. A particular version of VMAF called VMAF NEG has been designed to be more robust against such attacks and therefore it should be more useful for fine-tuning of codecs. In this paper, our contributions are threefold. First, we identify and analyze the still existing vulnerability of VMAF NEG towards attacks, particulary towards the attack that consists in employing VMAF NEG for image codec fine-tuning. Second, to benefit from VMAF NEG's high correlation with human perception, we propose a robust loss including VMAF NEG for fine-tuning either the encoder or the decoder. Third, we support our quantitative objective results by providing perceptive impressions of some image examples.