Video Coding Using Learned Latent GAN Compression

TL;DR

This paper introduces SGANC, a novel facial video compression method leveraging GANs and latent space inversion, achieving superior image and video quality at low bit rates compared to existing codecs.

Contribution

The paper presents a new GAN-based video compression paradigm using latent space inversion, normalizing flows, and a perceptual loss, improving efficiency and quality over prior methods.

Findings

Outperforms VTM, AV1, and recent deep learning codecs in quality.

Reduces perceptual distortion at low bit rates.

Faster training and simpler implementation.

Abstract

We propose in this paper a new paradigm for facial video compression. We leverage the generative capacity of GANs such as StyleGAN to represent and compress a video, including intra and inter compression. Each frame is inverted in the latent space of StyleGAN, from which the optimal compression is learned. To do so, a diffeomorphic latent representation is learned using a normalizing flows model, where an entropy model can be optimized for image coding. In addition, we propose a new perceptual loss that is more efficient than other counterparts. Finally, an entropy model for video inter coding with residual is also learned in the previously constructed latent representation. Our method (SGANC) is simple, faster to train, and achieves better results for image and video coding compared to state-of-the-art codecs such as VTM, AV1, and recent deep learning techniques. In particular, it drastically minimizes perceptual distortion at low bit rates.