Contextual Biasing for ASR in Speech LLM with Common Word Cues and Bias Word Position Prediction

TL;DR

This paper improves speech recognition accuracy for rare bias words by leveraging acoustic cues and bias word position prediction in speech-aware LLMs, without requiring phonetic expertise or G2P tools.

Contribution

It introduces a phoneme-free contextual biasing method using acoustic cues and a multi-output bias word position predictor, enhancing robustness and accuracy.

Findings

Reduces bias word recognition errors by 16.3%

Improves out-of-domain recognition accuracy

Eliminates need for phonetic knowledge or G2P tools

Abstract

Speech-aware LLMs (SLLMs) have recently achieved state-of-the-art ASR performance; however, they still fail to accurately transcribe bias words that appear rarely or never in the training data. Contextual biasing mechanisms are commonly implemented by introducing a predefined bias word list into the model via a text prompt or additional module. For further improvement, predefined bias words can be paired with their phoneme representations as pronunciation cues. Typically, phoneme sequences are generated through a G2P system that covers the target languages and domains of the bias words. Therefore, when a compatible G2P system is unavailable, phoneme-assisted contextual biasing becomes difficult to perform. Moreover, manually adding accurate phoneme sequences requires advanced phonetic knowledge. In this paper, we explore contextual biasing in SLLM based on acoustic cues associated with a set of common words whose pronunciations are partially similar to those of the target bias words. We assume ASR applications in which end users do not require special knowledge of phonetics or utilize G2P tools for inference. For enhanced robustness, we also introduce bias word positional prediction implemented in a multi-output learning fashion. Our method reduces bias word recognition errors by 16.3% compared to baseline systems, including on out-of-domain data.