ViBiDSampler: Enhancing Video Interpolation Using Bidirectional Diffusion Sampler

TL;DR

ViBiDSampler introduces a bidirectional diffusion sampling method that improves video keyframe interpolation by ensuring more coherent intermediate frames, achieving state-of-the-art quality and efficiency without extensive re-noising.

Contribution

The paper proposes a novel bidirectional sampling strategy for video interpolation that addresses off-manifold issues without re-noising, enhancing coherence and quality in generated videos.

Findings

Achieves state-of-the-art interpolation quality.

Interpolates 25 frames at 1024x576 in 195 seconds on a single GPU.

Effectively prevents artifacts common in previous methods.

Abstract

Recent progress in large-scale text-to-video (T2V) and image-to-video (I2V) diffusion models has greatly enhanced video generation, especially in terms of keyframe interpolation. However, current image-to-video diffusion models, while powerful in generating videos from a single conditioning frame, need adaptation for two-frame (start & end) conditioned generation, which is essential for effective bounded interpolation. Unfortunately, existing approaches that fuse temporally forward and backward paths in parallel often suffer from off-manifold issues, leading to artifacts or requiring multiple iterative re-noising steps. In this work, we introduce a novel, bidirectional sampling strategy to address these off-manifold issues without requiring extensive re-noising or fine-tuning. Our method employs sequential sampling along both forward and backward paths, conditioned on the start and end frames, respectively, ensuring more coherent and on-manifold generation of intermediate frames. Additionally, we incorporate advanced guidance techniques, CFG++ and DDS, to further enhance the interpolation process. By integrating these, our method achieves state-of-the-art performance, efficiently generating high-quality, smooth videos between keyframes. On a single 3090 GPU, our method can interpolate 25 frames at 1024 x 576 resolution in just 195 seconds, establishing it as a leading solution for keyframe interpolation.