Prompting with Phonemes: Enhancing LLMs' Multilinguality for Non-Latin Script Languages

TL;DR

This paper introduces a method that uses phonemic transcriptions to improve multilingual large language models' performance on non-Latin script languages by creating script-invariant representations, significantly closing the performance gap.

Contribution

The study proposes leveraging phonemic signals alongside orthographic scripts to enhance multilingual LLMs, especially for non-Latin scripts, and introduces a Mixed-ICL retrieval strategy for better in-context learning.

Findings

Phonemic signals improve non-Latin script language performance.

Mixed-ICL retrieval outperforms randomized retrieval.

Performance gains of up to 15.1% on non-Latin scripts.

Abstract

Although multilingual LLMs have achieved remarkable performance across benchmarks, we find they continue to underperform on non-Latin script languages across contemporary LLM families. This discrepancy arises from the fact that LLMs are pretrained with orthographic scripts, which are dominated by Latin characters that obscure their shared phonology with non-Latin scripts. We propose leveraging phonemic transcriptions as complementary signals to induce script-invariant representations. Our study demonstrates that integrating phonemic signals improves performance across both non-Latin and Latin script languages, with a particularly significant impact on closing the performance gap between the two. Through detailed experiments, we show that phonemic and orthographic scripts retrieve distinct examples for in-context learning (ICL). This motivates our proposed Mixed-ICL retrieval strategy, where further aggregation from both leads to our significant performance improvements for both Latin script languages (up to 12.6%) and non-Latin script languages (up to 15.1%) compared to randomized ICL retrieval.