mPLM-Sim: Better Cross-Lingual Similarity and Transfer in Multilingual Pretrained Language Models

TL;DR

This paper introduces mPLMSim, a novel language similarity measure derived from multilingual pretrained language models, which improves cross-lingual transfer by better selecting source languages based on learned language similarities.

Contribution

The paper proposes mPLMSim, a new similarity measure from mPLMs that correlates with linguistic similarities and enhances zero-shot cross-lingual transfer performance.

Findings

mPLM-Sim shows moderate correlation with linguistic similarity measures.

It outperforms traditional linguistic measures in selecting source languages.

Using mPLM-Sim improves zero-shot transfer accuracy by 1-2%.

Abstract

Recent multilingual pretrained language models (mPLMs) have been shown to encode strong language-specific signals, which are not explicitly provided during pretraining. It remains an open question whether it is feasible to employ mPLMs to measure language similarity, and subsequently use the similarity results to select source languages for boosting cross-lingual transfer. To investigate this, we propose mPLMSim, a language similarity measure that induces the similarities across languages from mPLMs using multi-parallel corpora. Our study shows that mPLM-Sim exhibits moderately high correlations with linguistic similarity measures, such as lexicostatistics, genealogical language family, and geographical sprachbund. We also conduct a case study on languages with low correlation and observe that mPLM-Sim yields more accurate similarity results. Additionally, we find that similarity results vary across different mPLMs and different layers within an mPLM. We further investigate whether mPLMSim is effective for zero-shot cross-lingual transfer by conducting experiments on both low-level syntactic tasks and high-level semantic tasks. The experimental results demonstrate that mPLM-Sim is capable of selecting better source languages than linguistic measures, resulting in a 1%-2% improvement in zero-shot cross-lingual transfer performance.