Alleviating the Inequality of Attention Heads for Neural Machine Translation

TL;DR

This paper introduces HeadMask, a simple masking method to address attention head inequality in Transformer models, leading to improved translation performance across multiple language pairs.

Contribution

It proposes a novel masking technique to balance attention heads in Transformer models, enhancing neural machine translation quality.

Findings

Translation quality improved on multiple language pairs.

Empirical analysis supports the effectiveness of HeadMask.

Addresses attention head imbalance in Transformer models.

Abstract

Recent studies show that the attention heads in Transformer are not equal. We relate this phenomenon to the imbalance training of multi-head attention and the model dependence on specific heads. To tackle this problem, we propose a simple masking method: HeadMask, in two specific ways. Experiments show that translation improvements are achieved on multiple language pairs. Subsequent empirical analyses also support our assumption and confirm the effectiveness of the method.