NERV++: An Enhanced Implicit Neural Video Representation

TL;DR

NeRV++ introduces an improved implicit neural video representation that enhances compression performance and reduces training complexity, making INR-based video coding more practical and competitive.

Contribution

The paper proposes NeRV++, an enhanced INR-based video codec with novel architectural modifications that significantly improve rate-distortion performance.

Findings

Achieves competitive compression results on multiple datasets.

Reduces training iterations and parameter count compared to previous INR methods.

Bridges the performance gap between INR-based and autoencoder-based video codecs.

Abstract

Neural fields, also known as implicit neural representations (INRs), have shown a remarkable capability of representing, generating, and manipulating various data types, allowing for continuous data reconstruction at a low memory footprint. Though promising, INRs applied to video compression still need to improve their rate-distortion performance by a large margin, and require a huge number of parameters and long training iterations to capture high-frequency details, limiting their wider applicability. Resolving this problem remains a quite challenging task, which would make INRs more accessible in compression tasks. We take a step towards resolving these shortcomings by introducing neural representations for videos NeRV++, an enhanced implicit neural video representation, as more straightforward yet effective enhancement over the original NeRV decoder architecture, featuring separable conv2d residual blocks (SCRBs) that sandwiches the upsampling block (UB), and a bilinear interpolation skip layer for improved feature representation. NeRV++ allows videos to be directly represented as a function approximated by a neural network, and significantly enhance the representation capacity beyond current INR-based video codecs. We evaluate our method on UVG, MCL JVC, and Bunny datasets, achieving competitive results for video compression with INRs. This achievement narrows the gap to autoencoder-based video coding, marking a significant stride in INR-based video compression research.