Contextual Biasing of Named-Entities with Large Language Models

TL;DR

This paper explores methods to improve speech recognition accuracy by using large language models with contextual biasing techniques like prompts, multi-task training, and dynamic prompting, achieving significant WER reductions.

Contribution

It introduces a prompt-based biasing approach, multi-task training, and dynamic prompting to enhance LLM-based rescoring in ASR without fine-tuning.

Findings

Biasing lists and few-shot examples improve WER by up to 17.8%.

Multi-task training yields a 20.0% WER reduction.

Dynamic prompting achieves an 11.3% WER improvement.

Abstract

This paper studies contextual biasing with Large Language Models (LLMs), where during second-pass rescoring additional contextual information is provided to a LLM to boost Automatic Speech Recognition (ASR) performance. We propose to leverage prompts for a LLM without fine tuning during rescoring which incorporate a biasing list and few-shot examples to serve as additional information when calculating the score for the hypothesis. In addition to few-shot prompt learning, we propose multi-task training of the LLM to predict both the entity class and the next token. To improve the efficiency for contextual biasing and to avoid exceeding LLMs' maximum sequence lengths, we propose dynamic prompting, where we select the most likely class using the class tag prediction, and only use entities in this class as contexts for next token prediction. Word Error Rate (WER) evaluation is performed on i) an internal calling, messaging, and dictation dataset, and ii) the SLUE-Voxpopuli dataset. Results indicate that biasing lists and few-shot examples can achieve 17.8% and 9.6% relative improvement compared to first pass ASR, and that multi-task training and dynamic prompting can achieve 20.0% and 11.3% relative WER improvement, respectively.