History-Guided Video Diffusion

TL;DR

This paper introduces DFoT, a novel video diffusion architecture with a training objective that supports flexible history conditioning, and proposes History Guidance methods that improve video quality, temporal consistency, and motion dynamics, enabling long and diverse video generation.

Contribution

The paper presents DFoT, a theoretically grounded architecture supporting variable-length history conditioning, and introduces History Guidance techniques that enhance video diffusion performance.

Findings

Significant improvement in video quality and temporal consistency with vanilla history guidance.

Enhanced motion dynamics and out-of-distribution generalization with advanced guidance methods.

Stable generation of extremely long videos using the proposed techniques.

Abstract

Classifier-free guidance (CFG) is a key technique for improving conditional generation in diffusion models, enabling more accurate control while enhancing sample quality. It is natural to extend this technique to video diffusion, which generates video conditioned on a variable number of context frames, collectively referred to as history. However, we find two key challenges to guiding with variable-length history: architectures that only support fixed-size conditioning, and the empirical observation that CFG-style history dropout performs poorly. To address this, we propose the Diffusion Forcing Transformer (DFoT), a video diffusion architecture and theoretically grounded training objective that jointly enable conditioning on a flexible number of history frames. We then introduce History Guidance, a family of guidance methods uniquely enabled by DFoT. We show that its simplest form, vanilla history guidance, already significantly improves video generation quality and temporal consistency. A more advanced method, history guidance across time and frequency further enhances motion dynamics, enables compositional generalization to out-of-distribution history, and can stably roll out extremely long videos. Project website: https://boyuan.space/history-guidance