Leveraging Large Language Models for Exploiting ASR Uncertainty

TL;DR

This paper demonstrates that prompting large language models with n-best ASR hypotheses improves speech intent classification and keyword spotting, effectively exploiting ASR uncertainty without changing core models.

Contribution

It introduces a method of using n-best ASR hypotheses as prompts for LLMs, enhancing speech understanding performance without modifying the underlying models.

Findings

n-best list prompts outperform 1-best hypotheses in speech tasks

Prompt engineering and fine-tuning improve LLM performance on spoken language understanding

Approach is effective on device-directed speech detection and keyword spotting

Abstract

While large language models excel in a variety of natural language processing (NLP) tasks, to perform well on spoken language understanding (SLU) tasks, they must either rely on off-the-shelf automatic speech recognition (ASR) systems for transcription, or be equipped with an in-built speech modality. This work focuses on the former scenario, where LLM's accuracy on SLU tasks is constrained by the accuracy of a fixed ASR system on the spoken input. Specifically, we tackle speech-intent classification task, where a high word-error-rate can limit the LLM's ability to understand the spoken intent. Instead of chasing a high accuracy by designing complex or specialized architectures regardless of deployment costs, we seek to answer how far we can go without substantially changing the underlying ASR and LLM, which can potentially be shared by multiple unrelated tasks. To this end, we propose prompting the LLM with an n-best list of ASR hypotheses instead of only the error-prone 1-best hypothesis. We explore prompt-engineering to explain the concept of n-best lists to the LLM; followed by the finetuning of Low-Rank Adapters on the downstream tasks. Our approach using n-best lists proves to be effective on a device-directed speech detection task as well as on a keyword spotting task, where systems using n-best list prompts outperform those using 1-best ASR hypothesis; thus paving the way for an efficient method to exploit ASR uncertainty via LLMs for speech-based applications.