Languages Transferred Within the Encoder: On Representation Transfer in Zero-Shot Multilingual Translation

TL;DR

This paper investigates how multilingual neural machine translation models transfer language representations within the encoder, revealing that transfer is language-specific rather than language-agnostic, and proposes methods to improve zero-shot translation performance.

Contribution

It introduces the identity pair concept for analysis and demonstrates that encoder transfer is language-specific, proposing two methods to enhance zero-shot translation.

Findings

Encoder transfers source language to target language subspace.

Zero-shot deficiency is due to entangled representations across languages.

Proposed methods significantly improve zero-shot translation performance.

Abstract

Understanding representation transfer in multilingual neural machine translation (MNMT) can reveal the reason for the zero-shot translation deficiency. In this work, we systematically analyze the representational issue of MNMT models. We first introduce the identity pair, translating a sentence to itself, to address the lack of the base measure in multilingual investigations, as the identity pair can reflect the representation of a language within the model. Then, we demonstrate that the encoder transfers the source language to the representational subspace of the target language instead of the language-agnostic state. Thus, the zero-shot translation deficiency arises because the representation of a translation is entangled with other languages and not transferred to the target language effectively. Based on our findings, we propose two methods: 1) low-rank language-specific embedding at the encoder, and 2) language-specific contrastive learning of the representation at the decoder. The experimental results on Europarl-15, TED-19, and OPUS-100 datasets show that our methods substantially enhance the performance of zero-shot translations without sacrifices in supervised directions by improving language transfer capacity, thereby providing practical evidence to support our conclusions. Codes are available at https://github.com/zhiqu22/ZeroTrans.