Low-Resource Domain Adaptation for Speech LLMs via Text-Only Fine-Tuning

TL;DR

This paper introduces a text-only fine-tuning method for Speech LLMs that enables effective domain adaptation in low-resource settings without additional audio data, maintaining performance and avoiding catastrophic forgetting.

Contribution

It presents a novel text-only fine-tuning approach with real-time evaluation to adapt Speech LLMs to new domains using unpaired text data.

Findings

Achieves competitive recognition performance on multiple datasets.

Maintains source-domain performance with minimal degradation.

Enhances generalization to new domains without catastrophic forgetting.

Abstract

Recent advances in automatic speech recognition (ASR) have combined speech encoders with large language models (LLMs) through projection, forming Speech LLMs with strong performance. However, adapting them to new domains remains challenging, especially in low-resource settings where paired speech-text data is scarce. We propose a text-only fine-tuning strategy for Speech LLMs using unpaired target-domain text without requiring additional audio. To preserve speech-text alignment, we introduce a real-time evaluation mechanism during fine-tuning. This enables effective domain adaptation while maintaining source-domain performance. Experiments on LibriSpeech, SlideSpeech, and Medical datasets show that our method achieves competitive recognition performance, with minimal degradation compared to full audio-text fine-tuning. It also improves generalization to new domains without catastrophic forgetting, highlighting the potential of text-only fine-tuning for low-resource domain adaptation of ASR.