Learning an Artificial Language for Knowledge-Sharing in Multilingual Translation

TL;DR

This paper introduces a discretized artificial language for multilingual neural translation, improving interpretability and robustness, and enabling analysis of language sharing effects.

Contribution

It proposes a novel discretization of encoder outputs into an artificial language, enhancing interpretability and robustness in multilingual translation models.

Findings

Discretization improves model robustness in unseen conditions.

Artificial language enables better analysis of model behavior.

Similar bridge languages enhance knowledge sharing.

Abstract

The cornerstone of multilingual neural translation is shared representations across languages. Given the theoretically infinite representation power of neural networks, semantically identical sentences are likely represented differently. While representing sentences in the continuous latent space ensures expressiveness, it introduces the risk of capturing of irrelevant features which hinders the learning of a common representation. In this work, we discretize the encoder output latent space of multilingual models by assigning encoder states to entries in a codebook, which in effect represents source sentences in a new artificial language. This discretization process not only offers a new way to interpret the otherwise black-box model representations, but, more importantly, gives potential for increasing robustness in unseen testing conditions. We validate our approach on large-scale experiments with realistic data volumes and domains. When tested in zero-shot conditions, our approach is competitive with two strong alternatives from the literature. We also use the learned artificial language to analyze model behavior, and discover that using a similar bridge language increases knowledge-sharing among the remaining languages.