RomanLens: The Role Of Latent Romanization In Multilinguality In LLMs

TL;DR

This paper investigates how Large Language Models utilize Latent Romanization, a process where non-Roman scripts are internally represented in Roman characters, to enhance multilingual understanding and translation capabilities.

Contribution

It introduces the concept of Latent Romanization, demonstrating its role in multilingual processing and providing mechanistic insights into how LLMs handle non-Roman scripts.

Findings

Intermediate layers often represent words in Romanized form before native script.

Semantic concepts are encoded similarly across native and Romanized scripts.

Romanized representations appear earlier in translation when the target language is Romanized.

Abstract

Large Language Models (LLMs) exhibit strong multilingual performance despite being predominantly trained on English-centric corpora. This raises a fundamental question: How do LLMs achieve such multilingual capabilities? Focusing on languages written in non-Roman scripts, we investigate the role of Romanization - the representation of non-Roman scripts using Roman characters - as a potential bridge in multilingual processing. Using mechanistic interpretability techniques, we analyze next-token generation and find that intermediate layers frequently represent target words in Romanized form before transitioning to native script, a phenomenon we term Latent Romanization. Further, through activation patching experiments, we demonstrate that LLMs encode semantic concepts similarly across native and Romanized scripts, suggesting a shared underlying representation. Additionally, for translation into non-Roman script languages, our findings reveal that when the target language is in Romanized form, its representations emerge earlier in the model's layers compared to native script. These insights contribute to a deeper understanding of multilingual representation in LLMs and highlight the implicit role of Romanization in facilitating language transfer.