Leveraging redundancy in attention with Reuse Transformers

TL;DR

This paper introduces a novel Transformer architecture that reuses attention scores across layers, reducing computation and memory while maintaining or improving performance.

Contribution

It systematically analyzes attention score redundancy and proposes reusing scores across layers to enhance efficiency without sacrificing accuracy.

Findings

Redundancy in attention scores across layers and heads.

Reusing attention scores maintains or improves performance.

Significant reduction in compute and memory usage.

Abstract

Pairwise dot product-based attention allows Transformers to exchange information between tokens in an input-dependent way, and is key to their success across diverse applications in language and vision. However, a typical Transformer model computes such pairwise attention scores repeatedly for the same sequence, in multiple heads in multiple layers. We systematically analyze the empirical similarity of these scores across heads and layers and find them to be considerably redundant, especially adjacent layers showing high similarity. Motivated by these findings, we propose a novel architecture that reuses attention scores computed in one layer in multiple subsequent layers. Experiments on a number of standard benchmarks show that reusing attention delivers performance equivalent to or better than standard transformers, while reducing both compute and memory usage.