Affine Transformation-Based Deep Frame Prediction

TL;DR

This paper introduces a neural network for deep frame prediction using affine transformations and adaptive filters, achieving smaller size and higher accuracy, leading to significant bit savings in video coding.

Contribution

The proposed model uniquely combines affine transformation estimation with adaptive spatial filters, reducing complexity and improving prediction accuracy in neural video coding.

Findings

Achieves approximately 7.3% bit savings in Low delay P configuration.

Outperforms prior neural networks in size and accuracy.

Integrates seamlessly with HEVC, enhancing compression efficiency.

Abstract

We propose a neural network model to estimate the current frame from two reference frames, using affine transformation and adaptive spatially-varying filters. The estimated affine transformation allows for using shorter filters compared to existing approaches for deep frame prediction. The predicted frame is used as a reference for coding the current frame. Since the proposed model is available at both encoder and decoder, there is no need to code or transmit motion information for the predicted frame. By making use of dilated convolutions and reduced filter length, our model is significantly smaller, yet more accurate, than any of the neural networks in prior works on this topic. Two versions of the proposed model - one for uni-directional, and one for bi-directional prediction - are trained using a combination of Discrete Cosine Transform (DCT)-based l1-loss with various transform sizes, multi-scale Mean Squared Error (MSE) loss, and an object context reconstruction loss. The trained models are integrated with the HEVC video coding pipeline. The experiments show that the proposed models achieve about 7.3%, 5.4%, and 4.2% bit savings for the luminance component on average in the Low delay P, Low delay, and Random access configurations, respectively.