Prompt-tuning in ASR systems for efficient domain-adaptation

TL;DR

This paper introduces prompt-tuning for transformer-based language models in ASR systems, enabling efficient domain adaptation with minimal parameters and achieving comparable performance to full fine-tuning.

Contribution

It proposes a prompt-tuning approach that adapts large language models for domain-specific ASR with significantly fewer parameters than traditional methods.

Findings

Prompt-tuning improves perplexity scores over unadapted LMs.

Parameter-efficient adaptation achieves results similar to full fine-tuning.

Enhanced domain-specific ASR performance demonstrated through WER reduction.

Abstract

Automatic Speech Recognition (ASR) systems have found their use in numerous industrial applications in very diverse domains. Since domain-specific systems perform better than their generic counterparts on in-domain evaluation, the need for memory and compute-efficient domain adaptation is obvious. Particularly, adapting parameter-heavy transformer-based language models used for rescoring ASR hypothesis is challenging. In this work, we overcome the problem using prompt-tuning, a methodology that trains a small number of domain token embedding parameters to prime a transformer-based LM to a particular domain. With just a handful of extra parameters per domain, we achieve much better perplexity scores over the baseline of using an unadapted LM. Despite being parameter-efficient, these improvements are comparable to those of fully-fine-tuned models with hundreds of millions of parameters. We replicate our findings in perplexity numbers to Word Error Rate in a domain-specific ASR system for one such domain.