Parameter Sharing Methods for Multilingual Self-Attentional Translation Models

TL;DR

This paper investigates parameter sharing strategies in multilingual Transformer models for neural machine translation, proposing partial sharing methods that improve translation quality across diverse language pairs.

Contribution

It introduces novel partial parameter sharing techniques that outperform full sharing, especially for linguistically diverse language pairs.

Findings

Full sharing boosts BLEU scores for similar languages.

Partial sharing significantly improves translation accuracy for different language families.

Proposed methods outperform full sharing in diverse multilingual settings.

Abstract

In multilingual neural machine translation, it has been shown that sharing a single translation model between multiple languages can achieve competitive performance, sometimes even leading to performance gains over bilingually trained models. However, these improvements are not uniform; often multilingual parameter sharing results in a decrease in accuracy due to translation models not being able to accommodate different languages in their limited parameter space. In this work, we examine parameter sharing techniques that strike a happy medium between full sharing and individual training, specifically focusing on the self-attentional Transformer model. We find that the full parameter sharing approach leads to increases in BLEU scores mainly when the target languages are from a similar language family. However, even in the case where target languages are from different families where full parameter sharing leads to a noticeable drop in BLEU scores, our proposed methods for partial sharing of parameters can lead to substantial improvements in translation accuracy.