Progressive Motion Context Refine Network for Efficient Video Frame Interpolation

TL;DR

The paper introduces PMCRNet, a lightweight and efficient video frame interpolation network that jointly predicts motion and image context, significantly reducing model size and inference time while maintaining high-quality results.

Contribution

It proposes a novel joint motion and context prediction framework with a simplified decoding process and a new multi-scale loss, improving efficiency over existing flow-based methods.

Findings

Achieves competitive interpolation quality on benchmarks.

Reduces model size and inference time significantly.

Maintains high accuracy with a simplified architecture.

Abstract

Recently, flow-based frame interpolation methods have achieved great success by first modeling optical flow between target and input frames, and then building synthesis network for target frame generation. However, above cascaded architecture can lead to large model size and inference delay, hindering them from mobile and real-time applications. To solve this problem, we propose a novel Progressive Motion Context Refine Network (PMCRNet) to predict motion fields and image context jointly for higher efficiency. Different from others that directly synthesize target frame from deep feature, we explore to simplify frame interpolation task by borrowing existing texture from adjacent input frames, which means that decoder in each pyramid level of our PMCRNet only needs to update easier intermediate optical flow, occlusion merge mask and image residual. Moreover, we introduce a new annealed multi-scale reconstruction loss to better guide the learning process of this efficient PMCRNet. Experiments on multiple benchmarks show that proposed approaches not only achieve favorable quantitative and qualitative results but also reduces current model size and running time significantly.