Content Adaptive based Motion Alignment Framework for Learned Video Compression

TL;DR

This paper introduces a content adaptive motion alignment framework for learned video compression that enhances performance by tailoring encoding strategies to content characteristics, leading to significant BD-rate savings.

Contribution

It proposes a novel two-stage flow-guided deformable warping, a multi-reference quality aware strategy, and a training-free module for improved motion compensation and content adaptation.

Findings

Achieves 24.95% BD-rate (PSNR) savings over baseline DCVC-TCM.

Outperforms traditional codecs like HM-16.25.

Demonstrates significant improvements on standard datasets.

Abstract

Recent advances in end-to-end video compression have shown promising results owing to their unified end-to-end learning optimization. However, such generalized frameworks often lack content-specific adaptation, leading to suboptimal compression performance. To address this, this paper proposes a content adaptive based motion alignment framework that improves performance by adapting encoding strategies to diverse content characteristics. Specifically, we first introduce a two-stage flow-guided deformable warping mechanism that refines motion compensation with coarse-to-fine offset prediction and mask modulation, enabling precise feature alignment. Second, we propose a multi-reference quality aware strategy that adjusts distortion weights based on reference quality, and applies it to hierarchical training to reduce error propagation. Third, we integrate a training-free module that downsamples frames by motion magnitude and resolution to obtain smooth motion estimation. Experimental results on standard test datasets demonstrate that our framework CAMA achieves significant improvements over state-of-the-art Neural Video Compression models, achieving a 24.95% BD-rate (PSNR) savings over our baseline model DCVC-TCM, while also outperforming reproduced DCVC-DC and traditional codec HM-16.25.