# Agglomerative Attention

## TL;DR

This paper introduces a linear-complexity attention mechanism for transformer networks, enabling scalable sequence modeling without sacrificing performance on language tasks.

## Contribution

It proposes a novel attention model with linear memory and computation requirements, maintaining competitive performance with traditional full attention models.

## Key findings

- Achieves comparable language modeling performance to full attention networks.
- Reduces memory and computation complexity from quadratic to linear.
- Enables training of larger models due to improved scalability.

## Abstract

Neural networks using transformer-based architectures have recently demonstrated great power and flexibility in modeling sequences of many types. One of the core components of transformer networks is the attention layer, which allows contextual information to be exchanged among sequence elements. While many of the prevalent network structures thus far have utilized full attention -- which operates on all pairs of sequence elements -- the quadratic scaling of this attention mechanism significantly constrains the size of models that can be trained. In this work, we present an attention model that has only linear requirements in memory and computation time. We show that, despite the simpler attention model, networks using this attention mechanism can attain comparable performance to full attention networks on language modeling tasks.