Romanization Encoding For Multilingual ASR

TL;DR

This paper presents a romanization encoding approach for multilingual ASR systems that significantly reduces vocabulary size and improves performance, especially in script-heavy languages and code-switching scenarios.

Contribution

It introduces a novel romanization encoding method combined with a balanced tokenizer and a FastConformer-RNNT framework, enhancing multilingual ASR flexibility and efficiency.

Findings

63.51% vocabulary reduction in Mandarin-English ASR

13.72% and 15.03% performance improvements on SEAME benchmarks

Effective handling of multiple script-heavy languages in ASR

Abstract

We introduce romanization encoding for script-heavy languages to optimize multilingual and code-switching Automatic Speech Recognition (ASR) systems. By adopting romanization encoding alongside a balanced concatenated tokenizer within a FastConformer-RNNT framework equipped with a Roman2Char module, we significantly reduce vocabulary and output dimensions, enabling larger training batches and reduced memory consumption. Our method decouples acoustic modeling and language modeling, enhancing the flexibility and adaptability of the system. In our study, applying this method to Mandarin-English ASR resulted in a remarkable 63.51% vocabulary reduction and notable performance gains of 13.72% and 15.03% on SEAME code-switching benchmarks. Ablation studies on Mandarin-Korean and Mandarin-Japanese highlight our method's strong capability to address the complexities of other script-heavy languages, paving the way for more versatile and effective multilingual ASR systems.