Spava: Accelerating Long-Video Understanding via Sequence-Parallelism-aware Approximate Attention

TL;DR

Spava is a sequence-parallel framework that accelerates long-video inference in large multimodal models by distributing approximate attention across multiple GPUs, significantly improving speed without performance loss.

Contribution

It introduces a novel sequence-parallel approximate attention method with system-level optimizations for efficient long-video processing in LMMs.

Findings

Achieves up to 12.72x speedup over existing methods.

Enables processing of longer, more complex videos.

Maintains performance while accelerating inference.

Abstract

The efficiency of long-video inference remains a critical bottleneck, mainly due to the dense computation in the prefill stage of Large Multimodal Models (LMMs). Existing methods either compress visual embeddings or apply sparse attention on a single GPU, yielding limited acceleration or degraded performance and restricting LMMs from handling longer, more complex videos. To overcome these issues, we propose Spava, a sequence-parallel framework with optimized attention that accelerates long-video inference across multiple GPUs. By distributing approximate attention, Spava reduces computation and increases parallelism, enabling efficient processing of more visual embeddings without compression and thereby improving task performance. System-level optimizations, such as load balancing and fused forward passes, further unleash the potential of Spava, delivering speedups of 12.72x, 1.70x, and 1.18x over FlashAttn, ZigZagRing, and APB, without notable performance loss. Code available at https://github.com/thunlp/APB