Bit Allocation using Optimization

TL;DR

This paper introduces a novel, optimal bit allocation method for neural video compression based on a new paradigm using Semi-Amortized Variational Inference, which outperforms existing algorithms in PSNR.

Contribution

It establishes a fundamental link between bit allocation in NVC and SAVI, extending SAVI to multi-level latent models, and proposes a practical approximation for improved performance.

Findings

Outperforms current bit allocation algorithms by approximately 0.5 dB PSNR.

Provides a theoretical bound on R-D performance of bit allocation.

Demonstrates the effectiveness of the SAVI-based approach in NVC scenarios.

Abstract

In this paper, we consider the problem of bit allocation in Neural Video Compression (NVC). First, we reveal a fundamental relationship between bit allocation in NVC and Semi-Amortized Variational Inference (SAVI). Specifically, we show that SAVI with GoP (Group-of-Picture)-level likelihood is equivalent to pixel-level bit allocation with precise rate \& quality dependency model. Based on this equivalence, we establish a new paradigm of bit allocation using SAVI. Different from previous bit allocation methods, our approach requires no empirical model and is thus optimal. Moreover, as the original SAVI using gradient ascent only applies to single-level latent, we extend the SAVI to multi-level such as NVC by recursively applying back-propagating through gradient ascent. Finally, we propose a tractable approximation for practical implementation. Our method can be applied to scenarios where performance outweights encoding speed, and serves as an empirical bound on the R-D performance of bit allocation. Experimental results show that current state-of-the-art bit allocation algorithms still have a room of $\approx 0.5$ dB PSNR to improve compared with ours. Code is available at \url{https://github.com/tongdaxu/Bit-Allocation-Using-Optimization}.