Speech Prefix-Tuning with RNNT Loss for Improving LLM Predictions

TL;DR

This paper introduces a speech prefix-tuning method using RNNT loss to enhance ASR performance with LLMs, achieving significant WER improvements without increasing model complexity.

Contribution

It proposes a novel speech prefix-tuning approach with RNNT loss and language-based soft prompting, improving ASR accuracy for both frozen and fine-tuned LLMs.

Findings

12% relative WER reduction with fine-tuned LLMs

31% relative WER reduction with frozen LLMs

Effective on 10 Indic languages

Abstract

In this paper, we focus on addressing the constraints faced when applying LLMs to ASR. Recent works utilize prefixLM-type models, which directly apply speech as a prefix to LLMs for ASR. We have found that optimizing speech prefixes leads to better ASR performance and propose applying RNNT loss to perform speech prefix-tuning. This is a simple approach and does not increase the model complexity or alter the inference pipeline. We also propose language-based soft prompting to further improve with frozen LLMs. Empirical analysis on realtime testset from 10 Indic languages demonstrate that our proposed speech prefix-tuning yields improvements with both frozen and fine-tuned LLMs. Our recognition results on an average of 10 Indics show that the proposed prefix-tuning with RNNT loss results in a 12\% relative improvement in WER over the baseline with a fine-tuned LLM. Our proposed approches with the frozen LLM leads to a 31\% relative improvement over basic soft-prompting prefixLM.