On the Rate-Distortion-Complexity Trade-offs of Neural Video Coding

TL;DR

This paper investigates the trade-offs between rate, distortion, and complexity in neural video coding, focusing on conditional autoencoders and residual coding methods to improve efficiency and performance.

Contribution

It analyzes how recent conditional residual coding approaches can better balance rate, distortion, and complexity compared to traditional methods.

Findings

Conditional residual coding offers improved trade-offs.

High-resolution features increase computational complexity.

Balance between coding efficiency and resource usage is achievable.

Abstract

This paper aims to delve into the rate-distortion-complexity trade-offs of modern neural video coding. Recent years have witnessed much research effort being focused on exploring the full potential of neural video coding. Conditional autoencoders have emerged as the mainstream approach to efficient neural video coding. The central theme of conditional autoencoders is to leverage both spatial and temporal information for better conditional coding. However, a recent study indicates that conditional coding may suffer from information bottlenecks, potentially performing worse than traditional residual coding. To address this issue, recent conditional coding methods incorporate a large number of high-resolution features as the condition signal, leading to a considerable increase in the number of multiply-accumulate operations, memory footprint, and model size. Taking DCVC as the common code base, we investigate how the newly proposed conditional residual coding, an emerging new school of thought, and its variants may strike a better balance among rate, distortion, and complexity.