Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion

TL;DR

Self Forcing presents a new training paradigm for autoregressive video diffusion models that reduces exposure bias and enables real-time high-quality video generation by conditioning on self-generated outputs during training.

Contribution

It introduces Self Forcing, a novel autoregressive training method with holistic sequence supervision, efficient KV caching, and stochastic gradient truncation for fast, high-quality video synthesis.

Findings

Achieves real-time streaming video generation with sub-second latency.

Matches or surpasses the quality of slower, non-causal diffusion models.

Effectively balances computational cost and performance with new training strategies.

Abstract

We introduce Self Forcing, a novel training paradigm for autoregressive video diffusion models. It addresses the longstanding issue of exposure bias, where models trained on ground-truth context must generate sequences conditioned on their own imperfect outputs during inference. Unlike prior methods that denoise future frames based on ground-truth context frames, Self Forcing conditions each frame's generation on previously self-generated outputs by performing autoregressive rollout with key-value (KV) caching during training. This strategy enables supervision through a holistic loss at the video level that directly evaluates the quality of the entire generated sequence, rather than relying solely on traditional frame-wise objectives. To ensure training efficiency, we employ a few-step diffusion model along with a stochastic gradient truncation strategy, effectively balancing computational cost and performance. We further introduce a rolling KV cache mechanism that enables efficient autoregressive video extrapolation. Extensive experiments demonstrate that our approach achieves real-time streaming video generation with sub-second latency on a single GPU, while matching or even surpassing the generation quality of significantly slower and non-causal diffusion models. Project website: http://self-forcing.github.io/