LLMs and Speech: Integration vs. Combination

TL;DR

This paper compares tight integration of pre-trained LLMs with acoustic models to traditional shallow fusion methods for speech recognition, analyzing various strategies and optimizations.

Contribution

It provides a comprehensive comparison of integration techniques, ablation studies, and optimization methods for combining LLMs with speech recognition models.

Findings

Tight integration improves recognition accuracy over shallow fusion.

Joint recognition with CTC reduces hallucinations in speech LLMs.

Fine-tuning LLMs on transcriptions enhances performance.

Abstract

In this work, we study how to best utilize pre-trained LLMs for automatic speech recognition. Specifically, we compare the tight integration of an acoustic model (AM) with the LLM ("speech LLM") to the traditional way of combining AM and LLM via shallow fusion. For tight integration, we provide ablations on the effect of different label units, fine-tuning strategies, LLM sizes and pre-training data, attention interfaces, encoder downsampling, text prompts, and length normalization. Additionally, we investigate joint recognition with a CTC model to mitigate hallucinations of speech LLMs and present effective optimizations for this joint recognition. For shallow fusion, we investigate the effect of fine-tuning the LLM on the transcriptions using different label units, and we compare rescoring AM hypotheses to single-pass recognition with label-wise or delayed fusion of AM and LLM scores. We train on Librispeech and Loquacious and evaluate our models on the HuggingFace ASR leaderboard.