Ride the Wave: Precision-Allocated Sparse Attention for Smooth Video Generation

TL;DR

PASA is a novel, training-free sparse attention framework that enhances efficiency and temporal smoothness in high-fidelity video generation by adaptive budgeting, hardware-aligned approximations, and stochastic routing.

Contribution

It introduces a dynamic, curvature-aware budgeting and stochastic attention routing to improve speed and stability in video diffusion models without additional training.

Findings

PASA significantly accelerates inference in video diffusion models.

It produces more fluid and stable video sequences compared to existing methods.

The approach effectively eliminates flickering caused by static sparsity patterns.

Abstract

Video Diffusion Transformers have revolutionized high-fidelity video generation but suffer from the massive computational burden of self-attention. While sparse attention provides a promising acceleration solution, existing methods frequently provoke severe visual flickering caused by static sparsity patterns and deterministic block routing. To resolve these limitations, we propose Precision-Allocated Sparse Attention (PASA), a training-free framework designed for highly efficient and temporally smooth video generation. First, we implement a curvature-aware dynamic budgeting mechanism. By profiling the generation trajectory acceleration across timesteps, we elastically allocate the exact-computation budget to secure high-precision processing strictly during critical semantic transitions. Second, we replace global homogenizing estimations with hardware-aligned grouped approximations, successfully capturing fine-grained local variations while maintaining peak compute throughput. Finally, we incorporate a stochastic selection bias into the attention routing mechanism. This probabilistic approach softens rigid selection boundaries and eliminates selection oscillation, effectively eradicating the localized computational starvation that drives temporal flickering. Extensive evaluations on leading video diffusion models demonstrate that PASA achieves substantial inference acceleration while consistently producing remarkably fluid and structurally stable video sequences.