DSV: Exploiting Dynamic Sparsity to Accelerate Large-Scale Video DiT Training

TL;DR

This paper introduces DSV, a method leveraging dynamic sparsity in attention to significantly accelerate large-scale video diffusion transformer training without sacrificing quality.

Contribution

DSV is a novel approach that captures dynamic sparsity patterns and employs custom kernels and parallelism to improve training efficiency of video DiTs.

Findings

Up to 3.02x higher training throughput

Scales to 128 GPUs and 520k token lengths

No loss in video quality

Abstract

Diffusion Transformers (DiTs) have shown remarkable performance in generating high-quality videos. However, the quadratic complexity of 3D full attention remains a bottleneck in scaling DiT training, especially with high-definition, lengthy videos, where it can consume up to 95% of processing time and demand specialized context parallelism. This paper introduces DSV to accelerate video DiT training by leveraging the dynamic attention sparsity we empirically observe. DSV uses a two-stage algorithm to capture the dynamic sparsity patterns via low-rank based approximation of the original query and key. It employs custom kernels to efficiently identify critical key-value pairs and compute the sparse attention. To accommodate the new sparsity dimension, DSV adopts a hybrid sparsity-aware context parallelism that re-balances the skewed workload across attention heads and blocks due to sparsity heterogeneity. DSV achieves up to 3.02x higher training throughput, scaling to 128 GPUs and 520k token lengths, without quality loss.