Generative Inbetweening through Frame-wise Conditions-Driven Video Generation

TL;DR

This paper introduces FCVG, a simple yet effective method for generative inbetweening that improves temporal stability in video frame interpolation by providing explicit frame-wise conditions, leading to more coherent and plausible videos.

Contribution

The paper proposes a novel Frame-wise Conditions-driven Video Generation (FCVG) approach that explicitly conditions each frame, enhancing temporal stability in video interpolation tasks.

Findings

FCVG significantly improves temporal stability across diverse scenarios.

The method works with both linear and non-linear interpolation curves.

Extensive evaluations show superior coherence compared to existing methods.

Abstract

Generative inbetweening aims to generate intermediate frame sequences by utilizing two key frames as input. Although remarkable progress has been made in video generation models, generative inbetweening still faces challenges in maintaining temporal stability due to the ambiguous interpolation path between two key frames. This issue becomes particularly severe when there is a large motion gap between input frames. In this paper, we propose a straightforward yet highly effective Frame-wise Conditions-driven Video Generation (FCVG) method that significantly enhances the temporal stability of interpolated video frames. Specifically, our FCVG provides an explicit condition for each frame, making it much easier to identify the interpolation path between two input frames and thus ensuring temporally stable production of visually plausible video frames. To achieve this, we suggest extracting matched lines from two input frames that can then be easily interpolated frame by frame, serving as frame-wise conditions seamlessly integrated into existing video generation models. In extensive evaluations covering diverse scenarios such as natural landscapes, complex human poses, camera movements and animations, existing methods often exhibit incoherent transitions across frames. In contrast, our FCVG demonstrates the capability to generate temporally stable videos using both linear and non-linear interpolation curves. Our project page and code are available at \url{https://fcvg-inbetween.github.io/}.