Auto-Encoding Variational Neural Machine Translation

TL;DR

This paper introduces a deep generative model for bilingual sentence pairs that jointly models source and target sentences, improving translation performance across various training scenarios using neural networks and variational inference.

Contribution

It presents a novel joint modeling approach for neural machine translation with efficient training and decoding methods, outperforming standard conditional models.

Findings

Joint model outperforms conditional models in all tested scenarios

Effective training with amortised variational inference

Fast approximate decoding method demonstrated

Abstract

We present a deep generative model of bilingual sentence pairs for machine translation. The model generates source and target sentences jointly from a shared latent representation and is parameterised by neural networks. We perform efficient training using amortised variational inference and reparameterised gradients. Additionally, we discuss the statistical implications of joint modelling and propose an efficient approximation to maximum a posteriori decoding for fast test-time predictions. We demonstrate the effectiveness of our model in three machine translation scenarios: in-domain training, mixed-domain training, and learning from a mix of gold-standard and synthetic data. Our experiments show consistently that our joint formulation outperforms conditional modelling (i.e. standard neural machine translation) in all such scenarios.