Lost in Transliteration: Bridging the Script Gap in Neural IR

TL;DR

This paper investigates the performance gap in neural IR systems when handling transliterated queries and proposes a fine-tuning approach on mixed native and Latinized texts to improve cross-script retrieval accuracy.

Contribution

It demonstrates that training on a mixture of native and transliterated scripts significantly enhances neural IR models' ability to handle transliterated queries effectively.

Findings

Models fine-tuned on mixed scripts perform nearly as well on transliterated queries as on native scripts.

Current models' performance drops sharply with transliterated queries, highlighting the script gap.

Transliterations can cause loss of query nuances, indicating the need for further research.

Abstract

Most human languages use scripts other than the Latin alphabet. Search users in these languages often formulate their information needs in a transliterated -- usually Latinized -- form for ease of typing. For example, Greek speakers might use Greeklish, and Arabic speakers might use Arabizi. This paper shows that current search systems, including those that use multilingual dense embeddings such as BGE-M3, do not generalise to this setting, and their performance rapidly deteriorates when exposed to transliterated queries. This creates a ``script gap" between the performance of the same queries when written in their native or transliterated form. We explore whether adapting the popular ``translate-train" paradigm to transliterations can enhance the robustness of multilingual Information Retrieval (IR) methods and bridge the gap between native and transliterated scripts. By exploring various combinations of non-Latin and Latinized query text for training, we investigate whether we can enhance the capacity of existing neural retrieval techniques and enable them to apply to this important setting. We show that by further fine-tuning IR models on an even mixture of native and Latinized text, they can perform this cross-script matching at nearly the same performance as when the query was formulated in the native script. Out-of-domain evaluation and further qualitative analysis show that transliterations can also cause queries to lose some of their nuances, motivating further research in this direction.