Cross-Lingual Ability of Multilingual Masked Language Models: A Study of Language Structure

TL;DR

This study investigates why multilingual masked language models like mBERT and XLM-R exhibit cross-lingual abilities, finding that language composition plays a more significant role than constituent order or word co-occurrence.

Contribution

The paper introduces an analysis method using artificial languages to identify which language properties influence cross-lingual transfer in multilingual models.

Findings

Language composition significantly impacts cross-lingual transfer.

Constituent order and word co-occurrence have limited effects.

Cross-lingual transfer depends more on shared language structures than on surface features.

Abstract

Multilingual pre-trained language models, such as mBERT and XLM-R, have shown impressive cross-lingual ability. Surprisingly, both of them use multilingual masked language model (MLM) without any cross-lingual supervision or aligned data. Despite the encouraging results, we still lack a clear understanding of why cross-lingual ability could emerge from multilingual MLM. In our work, we argue that cross-language ability comes from the commonality between languages. Specifically, we study three language properties: constituent order, composition and word co-occurrence. First, we create an artificial language by modifying property in source language. Then we study the contribution of modified property through the change of cross-language transfer results on target language. We conduct experiments on six languages and two cross-lingual NLP tasks (textual entailment, sentence retrieval). Our main conclusion is that the contribution of constituent order and word co-occurrence is limited, while the composition is more crucial to the success of cross-linguistic transfer.