HASTE: Training-Free Video Diffusion Acceleration via Head-Wise Adaptive Sparse Attention

TL;DR

HASTE introduces a training-free, head-wise adaptive sparse attention method for video diffusion models, significantly accelerating inference while preserving quality.

Contribution

It proposes a novel head-wise adaptive framework with mask reuse and error-guided calibration to improve speed-quality trade-offs in sparse attention.

Findings

Achieves up to 1.93x speedup at 720P resolution.

Maintains competitive video quality and similarity metrics.

Improves efficiency of pretrained video diffusion models.

Abstract

Diffusion-based video generation has advanced substantially in visual fidelity and temporal coherence, but practical deployment remains limited by the quadratic complexity of full attention. Training-free sparse attention is attractive because it accelerates pretrained models without retraining, yet existing online top-$p$ sparse attention still spends non-negligible cost on mask prediction and applies shared thresholds despite strong head-level heterogeneity. We show that these two overlooked factors limit the practical speed-quality trade-off of training-free sparse attention in Video DiTs. To address them, we introduce a head-wise adaptive framework with two plug-in components: Temporal Mask Reuse, which skips unnecessary mask prediction based on query-key drift, and Error-guided Budgeted Calibration, which assigns per-head top-$p$ thresholds by minimizing measured model-output error under a global sparsity budget. On Wan2.1-1.3B and Wan2.1-14B, our method consistently improves XAttention and SVG2, achieving up to 1.93 times speedup at 720P while maintaining competitive video quality and similarity metrics.