Speech Recognition Rescoring with Large Speech-Text Foundation Models

TL;DR

This paper introduces novel methods for using multi-modal speech-text foundation models to improve automatic speech recognition rescoring, achieving significant accuracy gains over existing models.

Contribution

It presents new techniques for applying multi-modal large language models to ASR rescoring and explores discriminative training to enhance performance.

Findings

Up to 20% relative improvement over Whisper large ASR

Up to 15% relative improvement over text-only LLM

Demonstrates cross-modal knowledge transfer benefits

Abstract

Large language models (LLM) have demonstrated the ability to understand human language by leveraging large amount of text data. Automatic speech recognition (ASR) systems are often limited by available transcribed speech data and benefit from a second pass rescoring using LLM. Recently multi-modal large language models, particularly speech and text foundational models have demonstrated strong spoken language understanding. Speech-Text foundational models leverage large amounts of unlabelled and labelled data both in speech and text modalities to model human language. In this work, we propose novel techniques to use multi-modal LLM for ASR rescoring. We also explore discriminative training to further improve the foundational model rescoring performance. We demonstrate cross-modal knowledge transfer in speech-text LLM can benefit rescoring. Our experiments demonstrate up-to 20% relative improvements over Whisper large ASR and up-to 15% relative improvements over text-only LLM.