Exploring Attention Map Reuse for Efficient Transformer Neural Networks

TL;DR

This paper investigates attention map reuse in Transformer models, demonstrating its effectiveness in accelerating inference and reducing computational costs for long sequences on CPU and GPU platforms.

Contribution

It provides a comprehensive analysis of attention map reuse, comparing it with other SA compression techniques and highlighting its advantages for long sequence processing.

Findings

Attention map reuse significantly speeds up inference.

It reduces memory and computation costs for long sequences.

Effective on both CPU and GPU platforms.

Abstract

Transformer-based deep neural networks have achieved great success in various sequence applications due to their powerful ability to model long-range dependency. The key module of Transformer is self-attention (SA) which extracts features from the entire sequence regardless of the distance between positions. Although SA helps Transformer performs particularly well on long-range tasks, SA requires quadratic computation and memory complexity with the input sequence length. Recently, attention map reuse, which groups multiple SA layers to share one attention map, has been proposed and achieved significant speedup for speech recognition models. In this paper, we provide a comprehensive study on attention map reuse focusing on its ability to accelerate inference. We compare the method with other SA compression techniques and conduct a breakdown analysis of its advantages for a long sequence. We demonstrate the effectiveness of attention map reuse by measuring the latency on both CPU and GPU platforms.