Mixture of Distributions Matters: Dynamic Sparse Attention for Efficient Video Diffusion Transformers

TL;DR

MOD-DiT introduces a dynamic, sampling-free attention method for video diffusion transformers that models evolving attention patterns efficiently, significantly improving generation speed and quality without costly sampling.

Contribution

It proposes a novel mixture-of-distribution framework for dynamic sparse attention, eliminating sampling and enhancing efficiency in video diffusion transformers.

Findings

Achieves faster video generation with maintained or improved quality.

Demonstrates consistent acceleration across multiple benchmarks.

Validates effectiveness over traditional sparse attention methods.

Abstract

While Diffusion Transformers (DiTs) have achieved notable progress in video generation, this long-sequence generation task remains constrained by the quadratic complexity inherent to self-attention mechanisms, creating significant barriers to practical deployment. Although sparse attention methods attempt to address this challenge, existing approaches either rely on oversimplified static patterns or require computationally expensive sampling operations to achieve dynamic sparsity, resulting in inaccurate pattern predictions and degraded generation quality. To overcome these limitations, we propose a \underline{\textbf{M}}ixture-\underline{\textbf{O}}f-\underline{\textbf{D}}istribution \textbf{DiT} (\textbf{MOD-DiT}), a novel sampling-free dynamic attention framework that accurately models evolving attention patterns through a two-stage process. First, MOD-DiT leverages prior information from early denoising steps and adopts a {distributed mixing approach} to model an efficient linear approximation model, which is then used to predict mask patterns for a specific denoising interval. Second, an online block masking strategy dynamically applies these predicted masks while maintaining historical sparsity information, eliminating the need for repetitive sampling operations. Extensive evaluations demonstrate consistent acceleration and quality improvements across multiple benchmarks and model architectures, validating MOD-DiT's effectiveness for efficient, high-quality video generation while overcoming the computational limitations of traditional sparse attention approaches.