Variational Neural Machine Translation

TL;DR

This paper introduces a variational neural machine translation model that incorporates a continuous latent variable to better capture source sentence semantics, leading to improved translation quality over traditional encoder-decoder models.

Contribution

The paper proposes a novel variational encoder-decoder framework for neural machine translation, enabling explicit semantic modeling and end-to-end training with efficient inference.

Findings

Significant improvements in Chinese-English translation accuracy.

Enhanced translation quality for English-German tasks.

Effective use of a neural posterior approximator with reparameterization.

Abstract

Models of neural machine translation are often from a discriminative family of encoderdecoders that learn a conditional distribution of a target sentence given a source sentence. In this paper, we propose a variational model to learn this conditional distribution for neural machine translation: a variational encoderdecoder model that can be trained end-to-end. Different from the vanilla encoder-decoder model that generates target translations from hidden representations of source sentences alone, the variational model introduces a continuous latent variable to explicitly model underlying semantics of source sentences and to guide the generation of target translations. In order to perform efficient posterior inference and large-scale training, we build a neural posterior approximator conditioned on both the source and the target sides, and equip it with a reparameterization technique to estimate the variational lower bound. Experiments on both Chinese-English and English- German translation tasks show that the proposed variational neural machine translation achieves significant improvements over the vanilla neural machine translation baselines.