Motion-Aware Generative Frame Interpolation

TL;DR

This paper introduces MoG, a novel motion-aware generative approach for frame interpolation that combines flow guidance with generative refinement to improve quality and stability in complex motion scenes.

Contribution

MoG uniquely integrates flow constraints with generative models, employing dual guidance and selective fine-tuning to enhance interpolation accuracy and artifact correction.

Findings

Outperforms state-of-the-art methods in quality and fidelity

Effective in complex motion regions and diverse scenarios

Bridges flow stability with generative flexibility

Abstract

Flow-based frame interpolation methods ensure motion stability through estimated intermediate flow but often introduce severe artifacts in complex motion regions. Recent generative approaches, boosted by large-scale pre-trained video generation models, show promise in handling intricate scenes. However, they frequently produce unstable motion and content inconsistencies due to the absence of explicit motion trajectory constraints. To address these challenges, we propose Motion-aware Generative frame interpolation (MoG) that synergizes intermediate flow guidance with generative capacities to enhance interpolation fidelity. Our key insight is to simultaneously enforce motion smoothness through flow constraints while adaptively correcting flow estimation errors through generative refinement. Specifically, we first introduce a dual guidance injection that propagates condition information using intermediate flow at both latent and feature levels, aligning the generated motion with flow-derived motion trajectories. Meanwhile, we implemented two critical designs, encoder-only guidance injection and selective parameter fine-tuning, which enable dynamic artifact correction in the complex motion regions. Extensive experiments on both real-world and animation benchmarks demonstrate that MoG outperforms state-of-the-art methods in terms of video quality and visual fidelity. Our work bridges the gap between flow-based stability and generative flexibility, offering a versatile solution for frame interpolation across diverse scenarios.