Making Every Head Count: Sparse Attention Without the Speed-Performance Trade-off

TL;DR

This paper introduces SPAttention, a novel sparse attention mechanism that reorganizes multi-head attention to reduce computational complexity from H*N^2 to N^2 without sacrificing performance, enabling more efficient large language models.

Contribution

The paper proposes Principled Structural Sparsity in attention, transforming multi-head attention into a collaborative, single computation, improving efficiency and model performance.

Findings

Reduces attention complexity from H*N^2 to N^2

Enhances computational efficiency without performance loss

Encourages head specialization for better dependency modeling

Abstract

The design of Large Language Models (LLMs) has long been hampered by a fundamental conflict within their core attention mechanism: its remarkable expressivity is built upon a computational complexity of O(H N^2) that grows quadratically with the context size (N) and linearly with the number of heads (H). This standard implementation harbors significant computational redundancy, as all heads independently compute attention over the same sequence space. Existing sparse methods, meanwhile, often trade information integrity for computational efficiency. To resolve this efficiency-performance trade-off, we propose SPAttention, whose core contribution is the introduction of a new paradigm we term Principled Structural Sparsity. SPAttention does not merely drop connections but instead reorganizes the computational task by partitioning the total attention workload into balanced, non-overlapping distance bands, assigning each head a unique segment. This approach transforms the multi-head attention mechanism from H independent O(N^2) computations into a single, collaborative O(N^2) computation, fundamentally reducing complexity by a factor of H. The structured inductive bias compels functional specialization among heads, enabling a more efficient allocation of computational resources from redundant modeling to distinct dependencies across the entire sequence span. Our work demonstrates that thoughtfully designed structural sparsity can serve as an effective inductive bias that simultaneously improves both computational efficiency and model performance, opening a new avenue for the architectural design of next-generation, high-performance LLMs.