Effective Text Adaptation for LLM-based ASR through Soft Prompt Fine-Tuning

TL;DR

This paper introduces a two-step soft prompt fine-tuning method for LLM-based ASR that significantly improves domain-specific transcription accuracy by effectively leveraging text data without losing domain knowledge.

Contribution

The proposed soft prompt fine-tuning strategy enhances domain adaptation in LLM-based ASR without compromising the model's domain-specific knowledge.

Findings

Achieved up to 9% WER reduction on target domain

Realized up to 18% EER reduction with the method

Further improvements when combined with domain-specific LM fusion

Abstract

The advent of Large Language Models (LLM) has reformed the Automatic Speech Recognition (ASR). Prompting LLM with audio embeddings to generate transcriptions becomes the new state-of-the-art ASR. Despite LLMs being trained with an extensive amount of text corpora, high-quality domain-specific text data can still significantly enhance ASR performance on domain adaptation tasks. Although LLM-based ASR can naturally incorporate more text corpora by fine-tuning the LLM decoder, fine-tuning such ASR on text-only data without paired prompts may diminish the effectiveness of domain-specific knowledge. To mitigate this issue, we propose a two-step soft prompt fine-tuning strategy that enhances domain-specific text adaptation. Experimental results show that text adaptation with our proposed method achieved a relative up to 9% Word Error Rate (WER) reduction and up to 18% Entity Error Rate (EER) reduction on the target domain compared to the baseline ASR. Combining this with domain-specific Language Model (LM) fusion can further improve the EER by a relative 2-5%