Head Forcing: Long Autoregressive Video Generation via Head Heterogeneity

TL;DR

Head Forcing introduces a novel, training-free approach to improve long-duration autoregressive video generation by assigning specialized cache strategies to different attention heads, significantly enhancing performance.

Contribution

It identifies functional roles of attention heads in AR video transformers and proposes a tailored cache strategy for each, extending generation duration without additional training.

Findings

Extends video generation from 5 seconds to over a minute.

Supports multi-prompt interactive synthesis.

Outperforms existing baselines in quality and duration.

Abstract

Autoregressive video diffusion models support real-time synthesis but suffer from error accumulation and context loss over long horizons. We discover that attention heads in AR video diffusion transformers serve functionally distinct roles as local heads for detail refinement, anchor heads for structural stabilization, and memory heads for long-range context aggregation, yet existing methods treat them uniformly, leading to suboptimal KV cache allocation. We propose Head Forcing, a training-free framework that assigns each head type a tailored KV cache strategy: local and anchor heads retain only essential tokens, while memory heads employ a hierarchical memory system with dynamic episodic updates for long-range consistency. A head-wise RoPE re-encoding scheme further ensures positional encodings remain within the pretrained range. Without additional training, Head Forcing extends generation from 5 seconds to minute-level duration, supports multi-prompt interactive synthesis, and consistently outperforms existing baselines. Project Page: https://jiahaotian-sjtu.github.io/headforcing.github.io/.