Adapting Image-to-Video Diffusion Models for Large-Motion Frame Interpolation

TL;DR

This paper introduces an adapted image-to-video diffusion model with a novel encoder and attention mechanism, significantly improving large-motion frame interpolation in videos.

Contribution

It proposes a conditional encoder, dual-branch feature extractor, and cross-frame attention to enhance large-motion frame interpolation capabilities.

Findings

Achieves superior FVD scores compared to state-of-the-art methods.

Effectively captures spatial and temporal information for accurate interpolation.

Excels in handling large motion scenarios in video frames.

Abstract

With the development of video generation models has advanced significantly in recent years, we adopt large-scale image-to-video diffusion models for video frame interpolation. We present a conditional encoder designed to adapt an image-to-video model for large-motion frame interpolation. To enhance performance, we integrate a dual-branch feature extractor and propose a cross-frame attention mechanism that effectively captures both spatial and temporal information, enabling accurate interpolations of intermediate frames. Our approach demonstrates superior performance on the Fr\'echet Video Distance (FVD) metric when evaluated against other state-of-the-art approaches, particularly in handling large motion scenarios, highlighting advancements in generative-based methodologies.