Towards Holistic Modeling for Video Frame Interpolation with Auto-regressive Diffusion Transformers

TL;DR

This paper introduces LDF-VFI, a holistic auto-regressive diffusion transformer for video frame interpolation that ensures long-range temporal coherence and generalizes to high resolutions, achieving state-of-the-art results.

Contribution

It presents a novel video-centric framework with a skip-concatenate sampling strategy and efficient long-sequence processing, advancing VFI performance and stability.

Findings

Achieves state-of-the-art results on VFI benchmarks.

Ensures long-range temporal coherence in video sequences.

Generalizes to arbitrary spatial resolutions like 4K.

Abstract

Existing video frame interpolation (VFI) methods often adopt a frame-centric approach, processing videos as independent short segments (e.g., triplets), which leads to temporal inconsistencies and motion artifacts. To overcome this, we propose a holistic, video-centric paradigm named Local Diffusion Forcing for Video Frame Interpolation (LDF-VFI). Our framework is built upon an auto-regressive diffusion transformer that models the entire video sequence to ensure long-range temporal coherence. To mitigate error accumulation inherent in auto-regressive generation, we introduce a novel skip-concatenate sampling strategy that effectively maintains temporal stability. Furthermore, LDF-VFI incorporates sparse, local attention and tiled VAE encoding, a combination that not only enables efficient processing of long sequences but also allows generalization to arbitrary spatial resolutions (e.g., 4K) at inference without retraining. An enhanced conditional VAE decoder, which leverages multi-scale features from the input video, further improves reconstruction fidelity. Empirically, LDF-VFI achieves state-of-the-art performance on challenging VFI benchmarks, demonstrating superior per-frame quality and temporal consistency, especially in scenes with large motion. The source code is available at https://github.com/xypeng9903/LDF-VFI.