NVRC: Neural Video Representation Compression

TL;DR

NVRC introduces a novel end-to-end INR-based video compression framework with advanced entropy coding and hierarchical model compression, achieving significant performance gains over standard codecs like VVC VTM.

Contribution

The paper presents the first INR-based video codec optimized end-to-end with new entropy coding and hierarchical model compression techniques.

Findings

Outperforms VVC VTM by 24% in PSNR on UVG dataset.

First INR-based video codec to achieve competitive performance.

End-to-end optimization of INR-based video compression.

Abstract

Recent advances in implicit neural representation (INR)-based video coding have demonstrated its potential to compete with both conventional and other learning-based approaches. With INR methods, a neural network is trained to overfit a video sequence, with its parameters compressed to obtain a compact representation of the video content. However, although promising results have been achieved, the best INR-based methods are still out-performed by the latest standard codecs, such as VVC VTM, partially due to the simple model compression techniques employed. In this paper, rather than focusing on representation architectures as in many existing works, we propose a novel INR-based video compression framework, Neural Video Representation Compression (NVRC), targeting compression of the representation. Based on the novel entropy coding and quantization models proposed, NVRC, for the first time, is able to optimize an INR-based video codec in a fully end-to-end manner. To further minimize the additional bitrate overhead introduced by the entropy models, we have also proposed a new model compression framework for coding all the network, quantization and entropy model parameters hierarchically. Our experiments show that NVRC outperforms many conventional and learning-based benchmark codecs, with a 24% average coding gain over VVC VTM (Random Access) on the UVG dataset, measured in PSNR. As far as we are aware, this is the first time an INR-based video codec achieving such performance. The implementation of NVRC will be released.