CMT-LLM: Contextual Multi-Talker ASR Utilizing Large Language Models

TL;DR

This paper introduces a unified ASR framework that combines multi-talker speech recognition and contextual biasing using pretrained speech encoders and large language models, improving recognition accuracy in complex scenarios.

Contribution

It presents a novel integrated approach that merges multi-talker recognition with contextual biasing, including a two-stage filtering algorithm for rare word incorporation.

Findings

Achieves 7.9% WER on LibriMix

Achieves 32.9% WER on AMI SDM

Outperforms traditional biasing methods

Abstract

In real-world applications, automatic speech recognition (ASR) systems must handle overlapping speech from multiple speakers and recognize rare words like technical terms. Traditional methods address multi-talker ASR and contextual biasing separately, limiting performance in complex scenarios. We propose a unified framework that combines multi-talker overlapping speech recognition and contextual biasing into a single task. Our ASR method integrates pretrained speech encoders and large language models (LLMs), using optimized finetuning strategies. We also introduce a two-stage filtering algorithm to efficiently identify relevant rare words from large biasing lists and incorporate them into the LLM's prompt input, enhancing rare word recognition. Experiments show that our approach outperforms traditional contextual biasing methods, achieving a WER of 7.9% on LibriMix and 32.9% on AMI SDM when the biasing size is 1,000, demonstrating its effectiveness in complex speech scenarios.