VecAttention: Vector-wise Sparse Attention for Accelerating Long Context Inference

TL;DR

VecAttention introduces a vector-wise sparse attention method that significantly accelerates long-context video inference while maintaining accuracy, outperforming existing sparse attention approaches.

Contribution

The paper proposes VecAttention, a novel vector-wise sparse attention framework that leverages vertical-vector patterns for better efficiency and accuracy in video models.

Findings

Achieves 2.65× speedup over full attention.

Attains 1.83× speedup over state-of-the-art sparse methods.

Maintains comparable accuracy to full attention.

Abstract

Long-context video understanding and generation pose a significant computational challenge for Transformer-based video models due to the quadratic complexity of self-attention. While existing sparse attention methods employ coarse-grained patterns to improve efficiency, they typically incur redundant computation and suboptimal performance. To address this issue, in this paper, we propose \textbf{VecAttention}, a novel framework of vector-wise sparse attention that achieves superior accuracy-efficiency trade-offs for video models. We observe that video attention maps exhibit a strong vertical-vector sparse pattern, and further demonstrate that this vertical-vector pattern offers consistently better accuracy-sparsity trade-offs compared with existing coarse-grained sparse patterns. Based on this observation, VecAttention dynamically selects and processes only informative vertical vectors through a lightweight important-vector selection that minimizes memory access overhead and an optimized kernel of vector sparse attention. Comprehensive evaluations on video understanding (VideoMME, LongVideoBench, and VCRBench) and generation (VBench) tasks show that VecAttention delivers a 2.65$\times$ speedup over full attention and a 1.83$\times$ speedup over state-of-the-art sparse attention methods, with comparable accuracy to full attention. Our code is available at https://github.com/anminliu/VecAttention.