LLM-based phoneme-to-grapheme for phoneme-based speech recognition

TL;DR

This paper introduces an LLM-based phoneme-to-grapheme decoding method for phoneme-based speech recognition, improving crosslingual ASR performance over traditional WFST-based systems by addressing information loss with novel training strategies.

Contribution

It proposes a new LLM-based decoding framework for phoneme-to-grapheme conversion in speech recognition, with innovative training techniques to mitigate information loss.

Findings

Outperforms WFST-based systems in Polish and German crosslingual ASR

Achieves 3.6% and 6.9% relative WER reductions respectively

Demonstrates effectiveness of LLM-based decoding with proposed training strategies

Abstract

In automatic speech recognition (ASR), phoneme-based multilingual pre-training and crosslingual fine-tuning is attractive for its high data efficiency and competitive results compared to subword-based models. However, Weighted Finite State Transducer (WFST) based decoding is limited by its complex pipeline and inability to leverage large language models (LLMs). Therefore, we propose LLM-based phoneme-to-grapheme (LLM-P2G) decoding for phoneme-based ASR, consisting of speech-to-phoneme (S2P) and phoneme-to-grapheme (P2G). A challenge is that there seems to have information loss in cascading S2P and P2G. To address this challenge, we propose two training strategies: data augmentation with noisy phonemes (DANP), and randomized top-$K$ marginalized (TKM) training and decoding. Our experimental results show that LLM-P2G outperforms WFST-based systems in crosslingual ASR for Polish and German, by relative WER reductions of 3.6% and 6.9% respectively.