Bridging Speech and Text: Enhancing ASR with Pinyin-to-Character Pre-training in LLMs

TL;DR

This paper introduces a novel Pinyin-to-Character pre-training method for large language models to improve Chinese speech recognition, achieving significant relative performance gains on the AISHELL-1 corpus.

Contribution

It proposes a new pre-training approach using Pinyin embeddings and fine-tuning with LoRA to enhance LLMs for ASR tasks involving Chinese speech.

Findings

9.5% relative improvement on AISHELL-1

19.0% relative improvement with auxiliary data

Effective integration of pronunciation features into LLMs

Abstract

The integration of large language models (LLMs) with pre-trained speech models has opened up new avenues in automatic speech recognition (ASR). While LLMs excel in multimodal understanding tasks, effectively leveraging their capabilities for ASR remains a significant challenge. This paper presents a novel training approach to enhance LLM performance in ASR tasks. We propose pre-training LLMs on Pinyin embedding sequences, which represent pronunciation features, to generate corresponding Chinese characters. This step enables the LLM to adapt to generating text from pronunciation features before encountering real speech data. Furthermore, we fine-tune the LoRA parameters to enhance the LLM's understanding of speech modality information. In AISHELL-1 corpus, our approach yields a 9.5% relative improvement in ASR tasks compared to the baseline without Pinyi-to-Character pre-training. Additionally, incorporating auxiliary text data for Pinyi-to-Character pre-training further boosts performance, achieving a 19.0% relative improvement.