Learning Hard Retrieval Decoder Attention for Transformers

TL;DR

This paper introduces a hard retrieval attention mechanism for Transformers that attends to only one token per head, significantly speeding up decoding without sacrificing translation quality.

Contribution

It proposes a novel hard retrieval attention method that replaces standard attention with a simple retrieval operation, improving decoding speed in Transformer models.

Findings

Decoding speed increased by 1.43 times

Translation quality preserved across multiple tasks

Efficient attention mechanism maintains performance

Abstract

The Transformer translation model is based on the multi-head attention mechanism, which can be parallelized easily. The multi-head attention network performs the scaled dot-product attention function in parallel, empowering the model by jointly attending to information from different representation subspaces at different positions. In this paper, we present an approach to learning a hard retrieval attention where an attention head only attends to one token in the sentence rather than all tokens. The matrix multiplication between attention probabilities and the value sequence in the standard scaled dot-product attention can thus be replaced by a simple and efficient retrieval operation. We show that our hard retrieval attention mechanism is 1.43 times faster in decoding, while preserving translation quality on a wide range of machine translation tasks when used in the decoder self- and cross-attention networks.