Reducing The Mismatch Between Marginal and Learned Distributions in Neural Video Compression

TL;DR

This paper investigates the discrepancy between marginal and learned distributions in neural video compression, evaluates the amortization gap in current models, and proposes a generic method to reduce this gap, improving compression efficiency.

Contribution

It introduces an evaluation of the amortization gap in neural video codecs and proposes a novel, efficient method to mitigate this gap, enhancing compression performance.

Findings

Amortization gap varies across state-of-the-art models.

The proposed method reduces the gap by 2-5%.

Compression efficiency improves without quality loss.

Abstract

During the last four years, we have witnessed the success of end-to-end trainable models for image compression. Compared to decades of incremental work, these machine learning (ML) techniques learn all the components of the compression technique, which explains their actual superiority. However, end-to-end ML models have not yet reached the performance of traditional video codecs such as VVC. Possible explanations can be put forward: lack of data to account for the temporal redundancy, or inefficiency of latent's density estimation in the neural model. The latter problem can be defined by the discrepancy between the latent's marginal distribution and the learned prior distribution. This mismatch, known as amortization gap of entropy model, enlarges the file size of compressed data. In this paper, we propose to evaluate the amortization gap for three state-of-the-art ML video compression methods. Second, we propose an efficient and generic method to solve the amortization gap and show that it leads to an improvement between $2\%$ to $5\%$ without impacting reconstruction quality.