CANeRV: Content Adaptive Neural Representation for Video Compression

TL;DR

CANeRV introduces a content adaptive neural network for video compression that dynamically optimizes structure at sequence, frame, and spatial levels, outperforming existing methods including H.266/VVC.

Contribution

It proposes a novel adaptive INR-based video compression framework with dynamic adjustments and hierarchical structural adaptation for improved performance.

Findings

Outperforms H.266/VVC and state-of-the-art INR methods on diverse datasets.

Effectively captures dynamic and structural information within videos.

Enhances detail restoration through hierarchical structural adaptation.

Abstract

Recent advances in video compression introduce implicit neural representation (INR) based methods, which effectively capture global dependencies and characteristics of entire video sequences. Unlike traditional and deep learning based approaches, INR-based methods optimize network parameters from a global perspective, resulting in superior compression potential. However, most current INR methods utilize a fixed and uniform network architecture across all frames, limiting their adaptability to dynamic variations within and between video sequences. This often leads to suboptimal compression outcomes as these methods struggle to capture the distinct nuances and transitions in video content. To overcome these challenges, we propose Content Adaptive Neural Representation for Video Compression (CANeRV), an innovative INR-based video compression network that adaptively conducts structure optimisation based on the specific content of each video sequence. To better capture dynamic information across video sequences, we propose a dynamic sequence-level adjustment (DSA). Furthermore, to enhance the capture of dynamics between frames within a sequence, we implement a dynamic frame-level adjustment (DFA). {Finally, to effectively capture spatial structural information within video frames, thereby enhancing the detail restoration capabilities of CANeRV, we devise a structure level hierarchical structural adaptation (HSA).} Experimental results demonstrate that CANeRV can outperform both H.266/VVC and state-of-the-art INR-based video compression techniques across diverse video datasets.