Robust Multiple Description Neural Video Codec with Masked Transformer for Dynamic and Noisy Networks

TL;DR

NeuralMDC introduces a neural video codec using masked transformers for multiple description coding, achieving high error resilience and efficient compression without complex motion prediction, suitable for unreliable networks.

Contribution

It presents a novel neural MDC codec leveraging bidirectional transformers for simplified design and improved loss resilience over traditional residual-based methods.

Findings

State-of-the-art loss resilience in neural video coding

Minimal reduction in compression efficiency

Outperforms existing residual-coding-based error-resilient codecs

Abstract

Multiple Description Coding (MDC) is a promising error-resilient source coding method that is particularly suitable for dynamic networks with multiple (yet noisy and unreliable) paths. However, conventional MDC video codecs suffer from cumbersome architectures, poor scalability, limited loss resilience, and lower compression efficiency. As a result, MDC has never been widely adopted. Inspired by the potential of neural video codecs, this paper rethinks MDC design. We propose a novel MDC video codec, NeuralMDC, demonstrating how bidirectional transformers trained for masked token prediction can vastly simplify the design of MDC video codec. To compress a video, NeuralMDC starts by tokenizing each frame into its latent representation and then splits the latent tokens to create multiple descriptions containing correlated information. Instead of using motion prediction and warping operations, NeuralMDC trains a bidirectional masked transformer to model the spatial-temporal dependencies of latent representations and predict the distribution of the current representation based on the past. The predicted distribution is used to independently entropy code each description and infer any potentially lost tokens. Extensive experiments demonstrate NeuralMDC achieves state-of-the-art loss resilience with minimal sacrifices in compression efficiency, significantly outperforming the best existing residual-coding-based error-resilient neural video codec.