On Biasing Transformer Attention Towards Monotonicity

TL;DR

This paper introduces a monotonicity loss for standard attention mechanisms to promote monotonic alignment in sequence-to-sequence tasks, showing mixed results with some improvements on RNNs and selective attention heads.

Contribution

The work proposes a new monotonicity loss compatible with standard attention, tested across multiple NLP tasks, and explores its effects on transformer and RNN models.

Findings

Achieves largely monotonic behavior in models.

Larger gains observed on RNN baselines.

Selective biasing of attention heads yields isolated improvements.

Abstract

Many sequence-to-sequence tasks in natural language processing are roughly monotonic in the alignment between source and target sequence, and previous work has facilitated or enforced learning of monotonic attention behavior via specialized attention functions or pretraining. In this work, we introduce a monotonicity loss function that is compatible with standard attention mechanisms and test it on several sequence-to-sequence tasks: grapheme-to-phoneme conversion, morphological inflection, transliteration, and dialect normalization. Experiments show that we can achieve largely monotonic behavior. Performance is mixed, with larger gains on top of RNN baselines. General monotonicity does not benefit transformer multihead attention, however, we see isolated improvements when only a subset of heads is biased towards monotonic behavior.