Accelerating Prefilling for Long-Context LLMs via Sparse Pattern Sharing

TL;DR

This paper introduces a novel sparse attention mechanism for long-context LLMs that shares accurate attention patterns across heads, significantly improving efficiency without sacrificing accuracy.

Contribution

The paper proposes a new sparse attention method that leverages inter-head pattern similarity to enhance speed and accuracy in long-context inference.

Findings

Achieves superior or comparable speedup to state-of-the-art methods.

Maintains high accuracy by capturing true attention dynamics.

Reduces full attention computations to a small subset of heads.

Abstract

Sparse attention methods exploit the inherent sparsity in attention to speed up the prefilling phase of long-context inference, mitigating the quadratic complexity of full attention computation. While existing sparse attention methods rely on predefined patterns or inaccurate estimations to approximate attention behavior, they often fail to fully capture the true dynamics of attention, resulting in reduced efficiency and compromised accuracy. Instead, we propose a highly accurate sparse attention mechanism that shares similar yet precise attention patterns across heads, enabling a more realistic capture of the dynamic behavior of attention. Our approach is grounded in two key observations: (1) attention patterns demonstrate strong inter-head similarity, and (2) this similarity remains remarkably consistent across diverse inputs. By strategically sharing computed accurate patterns across attention heads, our method effectively captures actual patterns while requiring full attention computation for only a small subset of heads. Comprehensive evaluations demonstrate that our approach achieves superior or comparable speedup relative to state-of-the-art methods while delivering the best overall accuracy.