All at Once: Temporally Adaptive Multi-Frame Interpolation with Advanced Motion Modeling

TL;DR

This paper introduces a true multi-frame video interpolation method using a pyramidal network and advanced motion modeling, achieving higher quality, consistency, and efficiency compared to iterative approaches.

Contribution

It presents a one-shot multi-frame interpolator with a pyramidal network, a novel flow estimation, and cubic motion model, outperforming state-of-the-art methods in quality and speed.

Findings

Outperforms current best method by 1.57dB PSNR.

8 times smaller model and 7.7 times faster.

Generates visually pleasing, temporally consistent frames.

Abstract

Recent advances in high refresh rate displays as well as the increased interest in high rate of slow motion and frame up-conversion fuel the demand for efficient and cost-effective multi-frame video interpolation solutions. To that regard, inserting multiple frames between consecutive video frames are of paramount importance for the consumer electronics industry. State-of-the-art methods are iterative solutions interpolating one frame at the time. They introduce temporal inconsistencies and clearly noticeable visual artifacts. Departing from the state-of-the-art, this work introduces a true multi-frame interpolator. It utilizes a pyramidal style network in the temporal domain to complete the multi-frame interpolation task in one-shot. A novel flow estimation procedure using a relaxed loss function, and an advanced, cubic-based, motion model is also used to further boost interpolation accuracy when complex motion segments are encountered. Results on the Adobe240 dataset show that the proposed method generates visually pleasing, temporally consistent frames, outperforms the current best off-the-shelf method by 1.57db in PSNR with 8 times smaller model and 7.7 times faster. The proposed method can be easily extended to interpolate a large number of new frames while remaining efficient because of the one-shot mechanism.