Generative error correction for code-switching speech recognition using large language models

TL;DR

This paper introduces a generative error correction approach using large language models to improve code-switching speech recognition accuracy by leveraging multiple hypotheses and a trainable adapter, especially effective in low-resource scenarios.

Contribution

It proposes a novel generative error correction method with LLMs and a trainable adapter for CS-ASR, shifting from traditional rescoring techniques and addressing data scarcity.

Findings

Significant reduction in mixed error rate (MER) for CS-ASR.

LLMs demonstrate high data efficiency for hypotheses-to-transcription learning.

The method outperforms traditional rescoring approaches.

Abstract

Code-switching (CS) speech refers to the phenomenon of mixing two or more languages within the same sentence. Despite the recent advances in automatic speech recognition (ASR), CS-ASR is still a challenging task ought to the grammatical structure complexity of the phenomenon and the data scarcity of specific training corpus. In this work, we propose to leverage large language models (LLMs) and lists of hypotheses generated by an ASR to address the CS problem. Specifically, we first employ multiple well-trained ASR models for N-best hypotheses generation, with the aim of increasing the diverse and informative elements in the set of hypotheses. Next, we utilize the LLMs to learn the hypotheses-to-transcription (H2T) mapping by adding a trainable low-rank adapter. Such a generative error correction (GER) method directly predicts the accurate transcription according to its expert linguistic knowledge and N-best hypotheses, resulting in a paradigm shift from the traditional language model rescoring or error correction techniques. Experimental evidence demonstrates that GER significantly enhances CS-ASR accuracy, in terms of reduced mixed error rate (MER). Furthermore, LLMs show remarkable data efficiency for H2T learning, providing a potential solution to the data scarcity problem of CS-ASR in low-resource languages.