Content-Adaptive Motion Rate Adaption for Learned Video Compression

TL;DR

This paper presents an online, content-adaptive motion rate adaptation method for learned video compression, improving rate-distortion performance by dynamically adjusting motion coding based on individual sequence content.

Contribution

It introduces an $oldsymbol{ extit{ extalpha}}$-map for spatially adaptive bit allocation and an online back-propagation scheme for inference-time optimization, enhancing learned video codecs.

Findings

Significant improvement in rate-distortion performance on complex motion sequences.

Effective adaptation of motion bit rate through the $ extit{ extalpha}$-map.

Enhanced codec performance with look-ahead mechanism.

Abstract

This paper introduces an online motion rate adaptation scheme for learned video compression, with the aim of achieving content-adaptive coding on individual test sequences to mitigate the domain gap between training and test data. It features a patch-level bit allocation map, termed the $\alpha$-map, to trade off between the bit rates for motion and inter-frame coding in a spatially-adaptive manner. We optimize the $\alpha$-map through an online back-propagation scheme at inference time. Moreover, we incorporate a look-ahead mechanism to consider its impact on future frames. Extensive experimental results confirm that the proposed scheme, when integrated into a conditional learned video codec, is able to adapt motion bit rate effectively, showing much improved rate-distortion performance particularly on test sequences with complicated motion characteristics.