Variational Neural Machine Translation with Normalizing Flows

TL;DR

This paper introduces a novel Variational Neural Machine Translation framework using normalizing flows to model latent variables more effectively within Transformer models, leading to improved translation accuracy.

Contribution

It extends VNMT to Transformer architectures with a flexible normalizing flow-based posterior, enhancing latent variable modeling in neural machine translation.

Findings

Significant performance improvements over baselines

Effective in both in-domain and out-of-domain translation tasks

Enhanced latent variable utilization in Transformer models

Abstract

Variational Neural Machine Translation (VNMT) is an attractive framework for modeling the generation of target translations, conditioned not only on the source sentence but also on some latent random variables. The latent variable modeling may introduce useful statistical dependencies that can improve translation accuracy. Unfortunately, learning informative latent variables is non-trivial, as the latent space can be prohibitively large, and the latent codes are prone to be ignored by many translation models at training time. Previous works impose strong assumptions on the distribution of the latent code and limit the choice of the NMT architecture. In this paper, we propose to apply the VNMT framework to the state-of-the-art Transformer and introduce a more flexible approximate posterior based on normalizing flows. We demonstrate the efficacy of our proposal under both in-domain and out-of-domain conditions, significantly outperforming strong baselines.