One Script Instead of Hundreds? On Pretraining Romanized Encoder Language Models

TL;DR

This study evaluates the impact of romanization on pretraining multilingual encoder models across diverse languages, finding minimal performance loss for segmental scripts and highlighting encoding efficiency benefits, with some degradation in morphosyllabic scripts.

Contribution

It provides a comprehensive analysis of romanization effects on high-resource languages during pretraining, addressing gaps in understanding its general applicability and impact on model performance.

Findings

Negligible performance loss for segmental scripts with romanization.

Degradation observed in morphosyllabic scripts mitigated by higher-fidelity romanization.

Romanization improves encoding efficiency without significant performance costs.

Abstract

Exposing latent lexical overlap, script romanization has emerged as an effective strategy for improving cross-lingual transfer (XLT) in multilingual language models (mLMs). Most prior work, however, focused on setups that favor romanization the most: (1) transfer from high-resource Latin-script to low-resource non-Latin-script languages and/or (2) between genealogically closely related languages with different scripts. It thus remains unclear whether romanization is a good representation choice for pretraining general-purpose mLMs, or, more precisely, if information loss associated with romanization harms performance for high-resource languages. We address this gap by pretraining encoder LMs from scratch on both romanized and original texts for six typologically diverse high-resource languages, investigating two potential sources of degradation: (i) loss of script-specific information and (ii) negative cross-lingual interference from increased vocabulary overlap. Using two romanizers with different fidelity profiles, we observe negligible performance loss for languages with segmental scripts, whereas languages with morphosyllabic scripts (Chinese and Japanese) suffer degradation that higher-fidelity romanization mitigates but cannot fully recover. Importantly, comparing monolingual LMs with their mLM counterpart, we find no evidence that increased subword overlap induces negative interference. We further show that romanization improves encoding efficiency (i.e., fertility) for segmental scripts at a negligible performance cost.