Parallel Context Modeling for Sliding Window Attention in Neural Video Coding

TL;DR

This paper introduces P-SWA, a parallel decoding method for neural video coding that significantly reduces latency and improves rate-distortion performance by leveraging diagonal wavefronts and enhanced context modeling.

Contribution

It presents a novel parallel decoding approach using diagonal wavefronts, embedding a hyperprior, and an accumulator to fuse information, addressing latency and performance issues in neural video codecs.

Findings

Decoding speed increased by 36% over parallel VCT.

Achieved up to 10.0% BD-rate savings for I-frames.

Achieved up to 7.1% BD-rate savings for P-frames.

Abstract

Most neural video codecs rely on temporal conditioning, which makes them susceptible to error propagation over long sequences. While Transformer-based architectures like the VCT offer a drift-free alternative, they suffer from high computational complexity and inferior RD performance. The recent SWA addresses these shortcomings by reducing complexity and enhancing RD performance, yet it restricts decoding to a strictly sequential raster-scan order, creating a critical bottleneck in decoding latency. To resolve this, we propose P-SWA, utilizing diagonal wavefronts to enable parallel decoding. By embedding a hyperprior and introducing an accumulator to fuse side information and local spatial context, our method increases decoding speed by 36% over the parallel VCT. Simultaneously, it achieves Bj{\o}ntegaard Delta-rate savings of up to 10.0% for I-frames and 7.1% for P-frames over the SWA baseline.