Breaking the Script Barrier in Multilingual Pre-Trained Language Models with Transliteration-Based Post-Training Alignment

TL;DR

This paper introduces a transliteration-based post-training alignment method to enhance cross-lingual transfer in multilingual models, especially for languages with different scripts, resulting in significant performance improvements.

Contribution

The paper proposes a novel transliteration-based post-pretraining alignment technique to improve cross-lingual transfer for languages with different scripts, demonstrating substantial gains in various tasks.

Findings

Models outperform original models by up to 50% after PPA.

Significant improvements when using non-English source languages.

Effective across diverse language groups and downstream tasks.

Abstract

Multilingual pre-trained models (mPLMs) have shown impressive performance on cross-lingual transfer tasks. However, the transfer performance is often hindered when a low-resource target language is written in a different script than the high-resource source language, even though the two languages may be related or share parts of their vocabularies. Inspired by recent work that uses transliteration to address this problem, our paper proposes a transliteration-based post-pretraining alignment (PPA) method aiming to improve the cross-lingual alignment between languages using diverse scripts. We select two areal language groups, $\textbf{Mediterranean-Amharic-Farsi}$ and $\textbf{South+East Asian Languages}$, wherein the languages are mutually influenced but use different scripts. We apply our method to these language groups and conduct extensive experiments on a spectrum of downstream tasks. The results show that after PPA, models consistently outperform the original model (up to 50% for some tasks) in English-centric transfer. In addition, when we use languages other than English as sources in transfer, our method obtains even larger improvements. We will make our code and models publicly available at \url{https://github.com/cisnlp/Transliteration-PPA}.