Video Frame Interpolation via Structure-Motion based Iterative Fusion

TL;DR

This paper introduces a novel structure-motion iterative fusion approach for video frame interpolation, combining the strengths of optical flow and kernel-based methods to produce sharper, more accurate interpolated frames with enhanced structural and motion consistency.

Contribution

It proposes an end-to-end learnable framework that fuses structure-based and motion-based interpolation with iterative refinement and saliency-aware evaluation, improving performance with less training data.

Findings

Outperforms state-of-the-art methods on three benchmarks.

Achieves superior metrics with only one-tenth of the training data.

Incorporates saliency masks to better handle foreground and background objects.

Abstract

Video Frame Interpolation synthesizes non-existent images between adjacent frames, with the aim of providing a smooth and consistent visual experience. Two approaches for solving this challenging task are optical flow based and kernel-based methods. In existing works, optical flow based methods can provide accurate point-to-point motion description, however, they lack constraints on object structure. On the contrary, kernel-based methods focus on structural alignment, which relies on semantic and apparent features, but tends to blur results. Based on these observations, we propose a structure-motion based iterative fusion method. The framework is an end-to-end learnable structure with two stages. First, interpolated frames are synthesized by structure-based and motion-based learning branches respectively, then, an iterative refinement module is established via spatial and temporal feature integration. Inspired by the observation that audiences have different visual preferences on foreground and background objects, we for the first time propose to use saliency masks in the evaluation processes of the task of video frame interpolation. Experimental results on three typical benchmarks show that the proposed method achieves superior performance on all evaluation metrics over the state-of-the-art methods, even when our models are trained with only one-tenth of the data other methods use.