PNVC: Towards Practical INR-based Video Compression

TL;DR

PNVC is a practical neural video codec that combines autoencoder and INR methods, achieving significant compression savings and real-time decoding speeds, advancing INR-based video coding towards deployment.

Contribution

Introduces PNVC, a novel INR-based video codec with structural reparameterization, hierarchical quality control, and scale-aware embeddings, outperforming existing INR codecs in efficiency and speed.

Findings

Achieves over 35% BD-rate savings against HEVC HM 18.0.

Maintains 20+ FPS decoding speed for 1080p videos.

Supports both low delay and random access configurations.

Abstract

Neural video compression has recently demonstrated significant potential to compete with conventional video codecs in terms of rate-quality performance. These learned video codecs are however associated with various issues related to decoding complexity (for autoencoder-based methods) and/or system delays (for implicit neural representation (INR) based models), which currently prevent them from being deployed in practical applications. In this paper, targeting a practical neural video codec, we propose a novel INR-based coding framework, PNVC, which innovatively combines autoencoder-based and overfitted solutions. Our approach benefits from several design innovations, including a new structural reparameterization-based architecture, hierarchical quality control, modulation-based entropy modeling, and scale-aware positional embedding. Supporting both low delay (LD) and random access (RA) configurations, PNVC outperforms existing INR-based codecs, achieving nearly 35%+ BD-rate savings against HEVC HM 18.0 (LD) - almost 10% more compared to one of the state-of-the-art INR-based codecs, HiNeRV and 5% more over VTM 20.0 (LD), while maintaining 20+ FPS decoding speeds for 1080p content. This represents an important step forward for INR-based video coding, moving it towards practical deployment. The source code will be available for public evaluation.