CWRNN-INVR: A Coupled WarpRNN based Implicit Neural Video Representation

TL;DR

This paper introduces CWRNN-INVR, a novel implicit neural video representation combining neural networks and residual grids, with a Coupled WarpRNN module for improved video reconstruction and compression.

Contribution

It proposes a mixed neural network and residual grid framework with a Coupled WarpRNN for explicit motion representation, enhancing video reconstruction quality.

Findings

Achieves the highest PSNR of 33.73 dB on UVG dataset among INVR methods.

Outperforms existing INVR methods in downstream tasks.

Demonstrates effective representation of regular and irregular video information.

Abstract

Implicit Neural Video Representation (INVR) has emerged as a novel approach for video representation and compression, using learnable grids and neural networks. Existing methods focus on developing new grid structures efficient for latent representation and neural network architectures with large representation capability, lacking the study on their roles in video representation. In this paper, the difference between INVR based on neural network and INVR based on grid is first investigated from the perspective of video information composition to specify their own advantages, i.e., neural network for general structure while grid for specific detail. Accordingly, an INVR based on mixed neural network and residual grid framework is proposed, where the neural network is used to represent the regular and structured information and the residual grid is used to represent the remaining irregular information in a video. A Coupled WarpRNN-based multi-scale motion representation and compensation module is specifically designed to explicitly represent the regular and structured information, thus terming our method as CWRNN-INVR. For the irregular information, a mixed residual grid is learned where the irregular appearance and motion information are represented together. The mixed residual grid can be combined with the coupled WarpRNN in a way that allows for network reuse. Experiments show that our method achieves the best reconstruction results compared with the existing methods, with an average PSNR of 33.73 dB on the UVG dataset under the 3M model and outperforms existing INVR methods in other downstream tasks. The code can be found at https://github.com/yiyang-sdu/CWRNN-INVR.git}{https://github.com/yiyang-sdu/CWRNN-INVR.git.