Boosting CTC-Based ASR Using LLM-Based Intermediate Loss Regularization

TL;DR

This paper introduces LAIL, a novel auxiliary loss framework that leverages large language models to improve CTC-based speech recognition, achieving better accuracy without sacrificing decoding speed.

Contribution

It proposes a new intermediate loss method that incorporates LLM knowledge into CTC models, enhancing linguistic modeling while maintaining efficiency.

Findings

Significant WER reduction on LibriSpeech, TEDLIUM2, WSJ datasets.

State-of-the-art performance for CTC-based ASR with minimal overhead.

Effective integration of LLM embeddings into CTC training process.

Abstract

End-to-end (E2E) automatic speech recognition (ASR) systems have revolutionized the field by integrating all components into a single neural network, with attention-based encoder-decoder models achieving state-of-the-art performance. However, their autoregressive decoding process limits inference speed, making them unsuitable for real-time applications. In contrast, CTC-based models offer faster, non-autoregressive decoding but struggle to model linguistic dependencies effectively. Addressing this challenge, we propose a novel auxiliary loss framework called Language-Aware Intermediate Loss (LAIL) to enhance CTC-based ASR using the linguistic knowledge of large language models (LLMs). By attaching connector layers to intermediate encoder layers, LAIL maps outputs to the embedding space of an LLM and computes a causal language modeling loss during training. This approach enhances linguistic modeling while preserving the computational efficiency of CTC decoding. Using the Conformer architecture and various LLaMA models, we demonstrate significant improvements in Word Error Rate (WER) on the LibriSpeech, TEDLIUM2, and WSJ corpora, achieving state-of-the-art performance for CTC-based ASR with minimal computational overhead.