Adding Interpretable Attention to Neural Translation Models Improves Word Alignment

TL;DR

This paper introduces an interpretable attention mechanism in neural translation models that improves word alignment accuracy by leveraging hidden representations and a novel inference method, achieving results comparable to traditional alignment tools.

Contribution

The authors propose a simple extension to the Transformer model that enables high-accuracy word alignment without requiring alignment supervision, using a new inference procedure based on stochastic gradient descent.

Findings

Alignments outperform naive attention interpretation methods.

Alignments are comparable to Giza++ on benchmark datasets.

The method works without explicit alignment training data.

Abstract

Multi-layer models with multiple attention heads per layer provide superior translation quality compared to simpler and shallower models, but determining what source context is most relevant to each target word is more challenging as a result. Therefore, deriving high-accuracy word alignments from the activations of a state-of-the-art neural machine translation model is an open challenge. We propose a simple model extension to the Transformer architecture that makes use of its hidden representations and is restricted to attend solely on encoder information to predict the next word. It can be trained on bilingual data without word-alignment information. We further introduce a novel alignment inference procedure which applies stochastic gradient descent to directly optimize the attention activations towards a given target word. The resulting alignments dramatically outperform the naive approach to interpreting Transformer attention activations, and are comparable to Giza++ on two publicly available data sets.