Towards a Common Understanding of Contributing Factors for Cross-Lingual Transfer in Multilingual Language Models: A Review

TL;DR

This review paper consolidates and analyzes existing research on the factors influencing the cross-lingual transfer capabilities of multilingual language models, providing a structured overview and guidance for future work.

Contribution

It categorizes and synthesizes empirical findings on factors affecting cross-lingual transfer in MLLMs, unifying diverse research streams and resolving conflicting results.

Findings

Identifies five key categories of contributing factors.

Summarizes empirical evidence and consensus among studies.

Provides guidance for future research and practical application.

Abstract

In recent years, pre-trained Multilingual Language Models (MLLMs) have shown a strong ability to transfer knowledge across different languages. However, given that the aspiration for such an ability has not been explicitly incorporated in the design of the majority of MLLMs, it is challenging to obtain a unique and straightforward explanation for its emergence. In this review paper, we survey literature that investigates different factors contributing to the capacity of MLLMs to perform zero-shot cross-lingual transfer and subsequently outline and discuss these factors in detail. To enhance the structure of this review and to facilitate consolidation with future studies, we identify five categories of such factors. In addition to providing a summary of empirical evidence from past studies, we identify consensuses among studies with consistent findings and resolve conflicts among contradictory ones. Our work contextualizes and unifies existing research streams which aim at explaining the cross-lingual potential of MLLMs. This review provides, first, an aligned reference point for future research and, second, guidance for a better-informed and more efficient way of leveraging the cross-lingual capacity of MLLMs.