Insights from Generative Modeling for Neural Video Compression

TL;DR

This paper explores neural video compression through generative modeling, introducing new architectures and methods that improve state-of-the-art performance by leveraging autoregressive transforms and structured priors.

Contribution

It presents novel neural video coding architectures inspired by generative models, including improved autoregressive transforms and entropy models, advancing the state-of-the-art in high-resolution video compression.

Findings

Achieved state-of-the-art compression performance on high-resolution videos.

Developed improved temporal autoregressive transforms and structured entropy models.

Introduced variable bitrate algorithms compatible with existing models.

Abstract

While recent machine learning research has revealed connections between deep generative models such as VAEs and rate-distortion losses used in learned compression, most of this work has focused on images. In a similar spirit, we view recently proposed neural video coding algorithms through the lens of deep autoregressive and latent variable modeling. We present these codecs as instances of a generalized stochastic temporal autoregressive transform, and propose new avenues for further improvements inspired by normalizing flows and structured priors. We propose several architectures that yield state-of-the-art video compression performance on high-resolution video and discuss their tradeoffs and ablations. In particular, we propose (i) improved temporal autoregressive transforms, (ii) improved entropy models with structured and temporal dependencies, and (iii) variable bitrate versions of our algorithms. Since our improvements are compatible with a large class of existing models, we provide further evidence that the generative modeling viewpoint can advance the neural video coding field.