Focused Forcing: Content-Aware Per-Frame KV Selection for Efficient Autoregressive Video Diffusion

TL;DR

Focused Forcing introduces a training-free method for selective key-value cache management in autoregressive video diffusion, enhancing efficiency and quality without additional training.

Contribution

It proposes a novel content-aware KV selection technique that dynamically focuses on relevant historical frames and heads, improving generation speed and quality.

Findings

Achieves up to 1.48x acceleration in autoregressive video generation.

Improves visual quality and text alignment without training.

Effectively manages KV caches across multiple paradigms.

Abstract

Recent advances in autoregressive video diffusion have enabled sequential and streaming video generation. However, long-horizon generation requires increasingly large KV caches, making efficient compression without sacrificing quality challenging. Existing methods mostly select historical frames based on attention scores, but their context decisions remain coarse. When multiple frames are generated in the same chunk, these methods often apply a shared history selection to the whole chunk, score historical frames solely by attention, and assign head-wise budgets either uniformly or by attention-pattern heuristics rather than explicit head-importance estimation. We show that frames within the same generated chunk can depend on distinct historical frames, that the same historical frame can receive different attention scores as its relative temporal distance to the current frames changes, and that masking different heads induces unequal generation degradation. Motivated by these findings, we propose \textbf{Focused Forcing}, a training-free KV selection method that focuses cached history along both generated-frame and head dimensions. For each generated frame, Focused Forcing preserves the most relevant and distinctive historical frames by combining attention scores with diversity scores of historical frames, while assigning larger budgets to heads with higher estimated importance. Across multiple autoregressive generation paradigms, Focused Forcing achieves up to $\textbf{1.48}\times$ end-to-end acceleration without training, while \textbf{improving visual quality and text alignment}. \textit{Our code will be released on GitHub.}