Does Transliteration Help Multilingual Language Modeling?

TL;DR

This paper investigates whether transliteration improves multilingual language models' performance, especially for low-resource languages with diverse scripts, by empirically evaluating on Indic languages and analyzing representation similarity.

Contribution

It provides the first empirical evidence that transliteration benefits low-resource languages in multilingual models without harming high-resource ones.

Findings

Transliteration improves performance for low-resource languages.

Transliteration increases cross-lingual representation similarity.

The effect of transliteration is statistically significant.

Abstract

Script diversity presents a challenge to Multilingual Language Models (MLLM) by reducing lexical overlap among closely related languages. Therefore, transliterating closely related languages that use different writing scripts to a common script may improve the downstream task performance of MLLMs. We empirically measure the effect of transliteration on MLLMs in this context. We specifically focus on the Indic languages, which have the highest script diversity in the world, and we evaluate our models on the IndicGLUE benchmark. We perform the Mann-Whitney U test to rigorously verify whether the effect of transliteration is significant or not. We find that transliteration benefits the low-resource languages without negatively affecting the comparatively high-resource languages. We also measure the cross-lingual representation similarity of the models using centered kernel alignment on parallel sentences from the FLORES-101 dataset. We find that for parallel sentences across different languages, the transliteration-based model learns sentence representations that are more similar.