Modeling Romanized Hindi and Bengali: Dataset Creation and Multilingual LLM Integration

TL;DR

This paper introduces a large transliteration dataset for Hindi and Bengali, and trains a multilingual seq2seq LLM that significantly improves transliteration accuracy for these languages.

Contribution

The paper creates a comprehensive transliteration dataset for Hindi and Bengali and develops a new multilingual LLM that outperforms existing models in transliteration tasks.

Findings

Significant BLEU and CER improvements over existing models

Nearly 1.8 million Hindi and 1 million Bengali transliteration pairs created

Pre-trained multilingual seq2seq LLM effectively handles Romanized scripts

Abstract

The development of robust transliteration techniques to enhance the effectiveness of transforming Romanized scripts into native scripts is crucial for Natural Language Processing tasks, including sentiment analysis, speech recognition, information retrieval, and intelligent personal assistants. Despite significant advancements, state-of-the-art multilingual models still face challenges in handling Romanized script, where the Roman alphabet is adopted to represent the phonetic structure of diverse languages. Within the South Asian context, where the use of Romanized script for Indo-Aryan languages is widespread across social media and digital communication platforms, such usage continues to pose significant challenges for cutting-edge multilingual models. While a limited number of transliteration datasets and models are available for Indo-Aryan languages, they generally lack sufficient diversity in pronunciation and spelling variations, adequate code-mixed data for large language model (LLM) training, and low-resource adaptation. To address this research gap, we introduce a novel transliteration dataset for two popular Indo-Aryan languages, Hindi and Bengali, which are ranked as the 3rd and 7th most spoken languages worldwide. Our dataset comprises nearly 1.8 million Hindi and 1 million Bengali transliteration pairs. In addition to that, we pre-train a custom multilingual seq2seq LLM based on Marian architecture using the developed dataset. Experimental results demonstrate significant improvements compared to existing relevant models in terms of BLEU and CER metrics.