Decipherment-Aware Multilingual Learning in Jointly Trained Language Models

TL;DR

This paper explores how multilingual learning in joint language models is fundamentally a decipherment process, emphasizing the importance of token alignment and decipherment settings for effective unsupervised multilingual learning.

Contribution

It establishes a connection between multilingual learning and decipherment, analyzing the impact of various factors and proposing lexical alignment to improve model performance.

Findings

Decipherment process is central to multilingual learning in joint models.

Token alignment significantly influences UCL performance.

Lexical alignment enhances downstream task results.

Abstract

The principle that governs unsupervised multilingual learning (UCL) in jointly trained language models (mBERT as a popular example) is still being debated. Many find it surprising that one can achieve UCL with multiple monolingual corpora. In this work, we anchor UCL in the context of language decipherment and show that the joint training methodology is a decipherment process pivotal for UCL. In a controlled setting, we investigate the effect of different decipherment settings on the multilingual learning performance and consolidate the existing opinions on the contributing factors to multilinguality. From an information-theoretic perspective we draw a limit to the UCL performance and demonstrate the importance of token alignment in challenging decipherment settings caused by differences in the data domain, language order and tokenization granularity. Lastly, we apply lexical alignment to mBERT and investigate the contribution of aligning different lexicon groups to downstream performance.