Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity

TL;DR

Sparse VideoGen significantly accelerates video diffusion transformers by exploiting inherent spatial-temporal sparsity in attention mechanisms, achieving over twofold speedup without sacrificing quality.

Contribution

It introduces a training-free framework that dynamically identifies sparse attention patterns and optimizes hardware implementation for efficient video generation.

Findings

Achieves up to 2.33x speedup on benchmark models.

Maintains high video quality despite acceleration.

Provides open-source code for practical use.

Abstract

Diffusion Transformers (DiTs) dominate video generation but their high computational cost severely limits real-world applicability, usually requiring tens of minutes to generate a few seconds of video even on high-performance GPUs. This inefficiency primarily arises from the quadratic computational complexity of 3D Full Attention with respect to the context length. In this paper, we propose a training-free framework termed Sparse VideoGen (SVG) that leverages the inherent sparsity in 3D Full Attention to boost inference efficiency. We reveal that the attention heads can be dynamically classified into two groups depending on distinct sparse patterns: (1) Spatial Head, where only spatially-related tokens within each frame dominate the attention output, and (2) Temporal Head, where only temporally-related tokens across different frames dominate. Based on this insight, SVG proposes an online profiling strategy to capture the dynamic sparse patterns and predicts the type of attention head. Combined with a novel hardware-efficient tensor layout transformation and customized kernel implementations, SVG achieves up to 2.28x and 2.33x end-to-end speedup on CogVideoX-v1.5 and HunyuanVideo, respectively, while preserving generation quality. Our code is open-sourced and is available at https://github.com/svg-project/Sparse-VideoGen