Versatile Transition Generation with Image-to-Video Diffusion

TL;DR

This paper introduces VTG, a versatile framework for generating smooth, high-quality video transitions from various inputs, addressing a gap in existing diffusion-based video generation methods.

Contribution

The paper proposes VTG, a novel transition generation framework with interpolation initialization and dual-directional fine-tuning, improving motion smoothness and fidelity in video transitions.

Findings

VTG outperforms existing methods on TransitBench across multiple tasks.

Interpolation-based initialization preserves object identity effectively.

Dual-directional motion fine-tuning enhances transition smoothness.

Abstract

Leveraging text, images, structure maps, or motion trajectories as conditional guidance, diffusion models have achieved great success in automated and high-quality video generation. However, generating smooth and rational transition videos given the first and last video frames as well as descriptive text prompts is far underexplored. We present VTG, a Versatile Transition video Generation framework that can generate smooth, high-fidelity, and semantically coherent video transitions. VTG introduces interpolation-based initialization that helps preserve object identity and handle abrupt content changes effectively. In addition, it incorporates dual-directional motion fine-tuning and representation alignment regularization to mitigate the limitations of pre-trained image-to-video diffusion models in motion smoothness and generation fidelity, respectively. To evaluate VTG and facilitate future studies on unified transition generation, we collected TransitBench, a comprehensive benchmark for transition generation covering two representative transition tasks: concept blending and scene transition. Extensive experiments show that VTG achieves superior transition performance consistently across all four tasks.