Fast-OMRA: Fast Online Motion Resolution Adaptation for Neural B-Frame Coding

TL;DR

Fast-OMRA introduces lightweight classifiers to efficiently select optimal frame downsampling factors for neural B-frame video coding, significantly reducing computation while maintaining high coding performance.

Contribution

It proposes novel classifier-based methods for rapid downsampling factor selection, addressing domain shift issues without re-training the entire codec.

Findings

Achieves comparable rate-distortion performance to brute-force methods.

Reduces computational complexity substantially.

Operates as an add-on without re-training the B-frame codec.

Abstract

Most learned B-frame codecs with hierarchical temporal prediction suffer from the domain shift issue caused by the discrepancy in the Group-of-Pictures (GOP) size used for training and test. As such, the motion estimation network may fail to predict large motion properly. One effective strategy to mitigate this domain shift issue is to downsample video frames for motion estimation. However, finding the optimal downsampling factor involves a time-consuming rate-distortion optimization process. This work introduces lightweight classifiers to determine the downsampling factor. To strike a good rate-distortion-complexity trade-off, our classifiers observe simple state signals, including only the coding and reference frames, to predict the best downsampling factor. We present two variants that adopt binary and multi-class classifiers, respectively. The binary classifier adopts the Focal Loss for training, classifying between motion estimation at high and low resolutions. Our multi-class classifier is trained with novel soft labels incorporating the knowledge of the rate-distortion costs of different downsampling factors. Both variants operate as add-on modules without the need to re-train the B-frame codec. Experimental results confirm that they achieve comparable coding performance to the brute-force search methods while greatly reducing computational complexity.