Neural Video Compression with Context Modulation

TL;DR

This paper introduces a novel neural video codec that enhances temporal context modeling through reference frame modulation, resulting in significant bitrate reductions compared to traditional and previous neural codecs.

Contribution

It proposes a two-step context modulation method using flow orientation and context compensation to better leverage reference information in neural video compression.

Findings

Achieves 22.7% bitrate reduction over H.266/VVC.

Offers 10.1% bitrate saving over previous state-of-the-art neural codec.

Effective elimination of irrelevant propagated information.

Abstract

Efficient video coding is highly dependent on exploiting the temporal redundancy, which is usually achieved by extracting and leveraging the temporal context in the emerging conditional coding-based neural video codec (NVC). Although the latest NVC has achieved remarkable progress in improving the compression performance, the inherent temporal context propagation mechanism lacks the ability to sufficiently leverage the reference information, limiting further improvement. In this paper, we address the limitation by modulating the temporal context with the reference frame in two steps. Specifically, we first propose the flow orientation to mine the inter-correlation between the reference frame and prediction frame for generating the additional oriented temporal context. Moreover, we introduce the context compensation to leverage the oriented context to modulate the propagated temporal context generated from the propagated reference feature. Through the synergy mechanism and decoupling loss supervision, the irrelevant propagated information can be effectively eliminated to ensure better context modeling. Experimental results demonstrate that our codec achieves on average 22.7% bitrate reduction over the advanced traditional video codec H.266/VVC, and offers an average 10.1% bitrate saving over the previous state-of-the-art NVC DCVC-FM. The code is available at https://github.com/Austin4USTC/DCMVC.