Extending Whisper with prompt tuning to target-speaker ASR

TL;DR

This paper introduces a prompt tuning method to adapt the Whisper ASR model for target-speaker recognition, achieving comparable performance to full fine-tuning with significantly fewer parameters.

Contribution

It proposes a parameter-efficient prompt tuning approach to extend Whisper for target-speaker ASR, reducing training costs and retaining original features.

Findings

Prompt tuning achieves state-of-the-art performance with only 1% of parameters.

Retains Whisper's features like inverse text normalization and timestamp tagging.

Significantly reduces training costs compared to full fine-tuning.

Abstract

Target-speaker automatic speech recognition (ASR) aims to transcribe the desired speech of a target speaker from multi-talker overlapped utterances. Most of the existing target-speaker ASR (TS-ASR) methods involve either training from scratch or fully fine-tuning a pre-trained model, leading to significant training costs and becoming inapplicable to large foundation models. This work leverages prompt tuning, a parameter-efficient fine-tuning approach, to extend Whisper, a large-scale single-talker ASR model, to TS-ASR. Variants of prompt tuning approaches along with their configurations are explored and optimized for TS-ASR.Experimental results show that prompt tuning can achieve performance comparable to state-of-the-art full training approaches while only requiring about 1\% of task-specific model parameters. Notably, the original Whisper's features, such as inverse text normalization and timestamp tagging, are retained in target-speaker ASR, keeping the generated transcriptions natural and informative.