LangSAMP: Language-Script Aware Multilingual Pretraining

TL;DR

LangSAMP introduces language and script embeddings into multilingual pretraining, significantly improving cross-lingual transfer and capturing language-specific nuances, demonstrated on a highly multilingual corpus.

Contribution

It proposes a novel method that incorporates language and script embeddings into Transformer models, enhancing multilingual representation learning and transfer performance.

Findings

Outperforms baseline in zero-shot crosslingual tasks

Captures language and script nuances effectively

Enables better source language selection for transfer

Abstract

Recent multilingual pretrained language models (mPLMs) often avoid using language embeddings -- learnable vectors assigned to individual languages. However, this places a significant burden on token representations to encode all language-specific information, which may hinder language neutrality. To address this limitation, we propose Language-Script Aware Multilingual Pretraining (LangSAMP), a method that incorporates both language and script embeddings to enhance representation learning. Specifically, we integrate these embeddings into the output of the Transformer blocks before passing the final representations to the language modeling head for prediction. We apply LangSAMP to the continual pretraining of XLM-R on a highly multilingual corpus covering more than 500 languages. The resulting model consistently outperforms the baseline in zero-shot crosslingual transfer across diverse downstream tasks. Extensive analysis reveals that language and script embeddings capture language- and script-specific nuances, which benefits more language-neutral representations, proven by improved pairwise cosine similarity. In our case study, we also show that language and script embeddings can be used to select better source languages for crosslingual transfer. We make our code and models publicly available at https://github.com/cisnlp/LangSAMP.