On the Choice of Perception Loss Function for Learned Video Compression

TL;DR

This paper investigates how different perception loss functions affect low-latency learned video compression, revealing trade-offs between temporal consistency and distortion, and proposing decoder-side choice of perception metrics for flexible reconstruction quality.

Contribution

It introduces a comparative analysis of two perception loss functions in learned video compression, showing that decoder-side selection can achieve near-universal performance.

Findings

PLF-JD preserves temporal correlation better but increases distortion.

PLF-FMD results in lower distortion but less temporal consistency.

Decoder-side choice of PLF can match the performance of trained systems without specific PLF optimization.

Abstract

We study causal, low-latency, sequential video compression when the output is subjected to both a mean squared-error (MSE) distortion loss as well as a perception loss to target realism. Motivated by prior approaches, we consider two different perception loss functions (PLFs). The first, PLF-JD, considers the joint distribution (JD) of all the video frames up to the current one, while the second metric, PLF-FMD, considers the framewise marginal distributions (FMD) between the source and reconstruction. Using information theoretic analysis and deep-learning based experiments, we demonstrate that the choice of PLF can have a significant effect on the reconstruction, especially at low-bit rates. In particular, while the reconstruction based on PLF-JD can better preserve the temporal correlation across frames, it also imposes a significant penalty in distortion compared to PLF-FMD and further makes it more difficult to recover from errors made in the earlier output frames. Although the choice of PLF decisively affects reconstruction quality, we also demonstrate that it may not be essential to commit to a particular PLF during encoding and the choice of PLF can be delegated to the decoder. In particular, encoded representations generated by training a system to minimize the MSE (without requiring either PLF) can be {\em near universal} and can generate close to optimal reconstructions for either choice of PLF at the decoder. We validate our results using (one-shot) information-theoretic analysis, detailed study of the rate-distortion-perception tradeoff of the Gauss-Markov source model as well as deep-learning based experiments on moving MNIST and KTH datasets.