Incorporating Structural Alignment Biases into an Attentional Neural Translation Model

TL;DR

This paper enhances neural machine translation models by integrating structural biases from traditional alignment models, leading to improved performance especially in low-resource language pairs.

Contribution

It introduces a method to incorporate structural alignment biases into attentional neural translation models, improving translation quality.

Findings

Improved translation accuracy over baseline models

Effective in low-resource language settings

Outperforms standard phrase-based models

Abstract

Neural encoder-decoder models of machine translation have achieved impressive results, rivalling traditional translation models. However their modelling formulation is overly simplistic, and omits several key inductive biases built into traditional models. In this paper we extend the attentional neural translation model to include structural biases from word based alignment models, including positional bias, Markov conditioning, fertility and agreement over translation directions. We show improvements over a baseline attentional model and standard phrase-based model over several language pairs, evaluating on difficult languages in a low resource setting.