The Impact of Vocabulary Overlaps on Knowledge Transfer in Multilingual Machine Translation

TL;DR

This paper investigates how vocabulary overlap affects knowledge transfer in multilingual neural machine translation, emphasizing the roles of language relatedness and domain match over vocabulary sharing.

Contribution

It provides systematic experiments comparing joint and disjoint vocabularies, highlighting the greater importance of language relatedness and domain match.

Findings

Vocabulary overlap improves transfer for related languages.

Domain match and language relatedness are more crucial than vocabulary overlap.

Disjoint vocabularies can still achieve effective transfer under certain conditions.

Abstract

Knowledge transfer, especially across related languages, has been found beneficial for multilingual neural machine translation (MNMT), but some aspects are still under-explored and deserve further investigation. A joint vocabulary is most often applied to form a uniform word embedding space, but since the impact of a disjoint vocabulary on model performance is far less studied, there is no consensus on how much knowledge transfer is mainly due to vocabulary overlap. In this paper, we present systematic experiments with joint and disjoint vocabularies, and auxiliary languages related and unrelated to the source language. We design this experiment in an out-of-domain setup in order to emphasize transfer and the impact of the auxiliary language. As expected, we yield better results with more extensive vocabulary overlaps typical for related languages, but our experiments also show that domain-match and language relatedness are more important than a joint vocabulary.