Curse of High Dimensionality Issue in Transformer for Long-context Modeling

TL;DR

This paper identifies the redundancy in attention mechanisms of transformers for long-context modeling, reformulates the problem to optimize attention, and proposes Dynamic Group Attention to reduce computational costs while maintaining performance.

Contribution

It introduces a novel reformulation of sequence modeling as supervised learning, develops a group coding strategy, and proposes Dynamic Group Attention to improve efficiency in long-context transformers.

Findings

DGA reduces computational costs significantly.

DGA maintains competitive performance.

Theoretical analysis confirms robustness and efficiency improvements.

Abstract

Transformer-based large language models (LLMs) excel in natural language processing tasks by capturing long-range dependencies through self-attention mechanisms. However, long-context modeling faces significant computational inefficiencies due to \textit{redundant} attention computations: while attention weights are often \textit{sparse}, all tokens consume \textit{equal} computational resources. In this paper, we reformulate traditional probabilistic sequence modeling as a \textit{supervised learning task}, enabling the separation of relevant and irrelevant tokens and providing a clearer understanding of redundancy. Based on this reformulation, we theoretically analyze attention sparsity, revealing that only a few tokens significantly contribute to predictions. Building on this, we formulate attention optimization as a linear coding problem and propose a \textit{group coding strategy}, theoretically showing its ability to improve robustness against random noise and enhance learning efficiency. Motivated by this, we propose \textit{Dynamic Group Attention} (DGA), which leverages the group coding to explicitly reduce redundancy by aggregating less important tokens during attention computation. Empirical results show that our DGA significantly reduces computational costs while maintaining competitive performance.Code is available at https://github.com/bolixinyu/DynamicGroupAttention.