Long-term Video Frame Interpolation via Feature Propagation

TL;DR

This paper introduces a novel feature propagation framework for long-term video frame interpolation that effectively handles large temporal gaps by propagating features instead of relying solely on motion estimation.

Contribution

The paper proposes a propagation network (PNet) with a motion-to-feature approach to improve long-term VFI, addressing limitations of traditional motion estimation methods.

Findings

Outperforms state-of-the-art methods on benchmark datasets.

Effectively handles large temporal gaps in video sequences.

Demonstrates robustness in long-term frame interpolation.

Abstract

Video frame interpolation (VFI) works generally predict intermediate frame(s) by first estimating the motion between inputs and then warping the inputs to the target time with the estimated motion. This approach, however, is not optimal when the temporal distance between the input sequence increases as existing motion estimation modules cannot effectively handle large motions. Hence, VFI works perform well for small frame gaps and perform poorly as the frame gap increases. In this work, we propose a novel framework to address this problem. We argue that when there is a large gap between inputs, instead of estimating imprecise motion that will eventually lead to inaccurate interpolation, we can safely propagate from one side of the input up to a reliable time frame using the other input as a reference. Then, the rest of the intermediate frames can be interpolated using standard approaches as the temporal gap is now narrowed. To this end, we propose a propagation network (PNet) by extending the classic feature-level forecasting with a novel motion-to-feature approach. To be thorough, we adopt a simple interpolation model along with PNet as our full model and design a simple procedure to train the full model in an end-to-end manner. Experimental results on several benchmark datasets confirm the effectiveness of our method for long-term VFI compared to state-of-the-art approaches.