On Benefits and Challenges of Conditional Interframe Video Coding in Light of Information Theory

TL;DR

This paper analyzes the theoretical advantages and practical challenges of conditional interframe video coding, highlighting when it outperforms residual coding and how bottlenecks can negate these benefits.

Contribution

It provides an information theoretical framework for understanding when conditional coding offers gains over residual coding in video compression.

Findings

Conditional coding can outperform residual coding under certain information conditions.

Information bottlenecks in the prediction signal can negate the benefits of conditional coding.

Theoretical analysis clarifies when and why conditional coding is advantageous.

Abstract

The rise of variational autoencoders for image and video compression has opened the door to many elaborate coding techniques. One example here is the possibility of conditional interframe coding. Here, instead of transmitting the residual between the original frame and the predicted frame (often obtained by motion compensation), the current frame is transmitted under the condition of knowing the prediction signal. In practice, conditional coding can be straightforwardly implemented using a conditional autoencoder, which has also shown good results in recent works. In this paper, we provide an information theoretical analysis of conditional coding for inter frames and show in which cases gains compared to traditional residual coding can be expected. We also show the effect of information bottlenecks which can occur in practical video coders in the prediction signal path due to the network structure, as a consequence of the data-processing theorem or due to quantization. We demonstrate that conditional coding has theoretical benefits over residual coding but that there are cases in which the benefits are quickly canceled by small information bottlenecks of the prediction signal.