VMonarch: Efficient Video Diffusion Transformers with Structured Attention

TL;DR

VMonarch introduces a structured attention mechanism using Monarch matrices for Video Diffusion Transformers, significantly reducing computational complexity and enabling efficient long-video processing with comparable or better quality.

Contribution

The paper proposes VMonarch, a novel structured attention method with Monarch matrices that achieves sub-quadratic complexity and improves efficiency over existing sparse attention techniques.

Findings

Reduces attention FLOPs by 17.5 times

Achieves over 5x speedup in attention computation for long videos

Maintains or improves generation quality compared to full attention

Abstract

The quadratic complexity of the attention mechanism severely limits the context scalability of Video Diffusion Transformers (DiTs). We find that the highly sparse spatio-temporal attention patterns exhibited in Video DiTs can be naturally represented by the Monarch matrix. It is a class of structured matrices with flexible sparsity, enabling sub-quadratic attention via an alternating minimization algorithm. Accordingly, we propose VMonarch, a novel attention mechanism for Video DiTs that enables efficient computation over the dynamic sparse patterns with structured Monarch matrices. First, we adapt spatio-temporal Monarch factorization to explicitly capture the intra-frame and inter-frame correlations of the video data. Second, we introduce a recomputation strategy to mitigate artifacts arising from instabilities during alternating minimization of Monarch matrices. Third, we propose a novel online entropy algorithm fused into FlashAttention, enabling fast Monarch matrix updates for long sequences. Extensive experiments demonstrate that VMonarch achieves comparable or superior generation quality to full attention on VBench after minimal tuning. It overcomes the attention bottleneck in Video DiTs, reduces attention FLOPs by a factor of 17.5, and achieves a speedup of over 5x in attention computation for long videos, surpassing state-of-the-art sparse attention methods at 90% sparsity.