Augmented Deep Contexts for Spatially Embedded Video Coding

TL;DR

This paper introduces SEVC, a novel neural video codec that uses spatially embedded references and a spatial-guided latent prior to better handle large motions and emerging objects, achieving superior rate-distortion performance.

Contribution

The paper proposes a spatially embedded video codec with augmented motion vectors and a spatial-guided latent prior, addressing limitations of temporal-only neural video codecs.

Findings

Reduces bitrate by 11.9% over state-of-the-art NVC.

Effectively handles large motions and emerging objects.

Enriches prior information with spatial-guided latent prior.

Abstract

Most Neural Video Codecs (NVCs) only employ temporal references to generate temporal-only contexts and latent prior. These temporal-only NVCs fail to handle large motions or emerging objects due to limited contexts and misaligned latent prior. To relieve the limitations, we propose a Spatially Embedded Video Codec (SEVC), in which the low-resolution video is compressed for spatial references. Firstly, our SEVC leverages both spatial and temporal references to generate augmented motion vectors and hybrid spatial-temporal contexts. Secondly, to address the misalignment issue in latent prior and enrich the prior information, we introduce a spatial-guided latent prior augmented by multiple temporal latent representations. At last, we design a joint spatial-temporal optimization to learn quality-adaptive bit allocation for spatial references, further boosting rate-distortion performance. Experimental results show that our SEVC effectively alleviates the limitations in handling large motions or emerging objects, and also reduces 11.9% more bitrate than the previous state-of-the-art NVC while providing an additional low-resolution bitstream. Our code and model are available at https://github.com/EsakaK/SEVC.