Perceiver-Prompt: Flexible Speaker Adaptation in Whisper for Chinese Disordered Speech Recognition

TL;DR

This paper presents Perceiver-Prompt, a novel speaker adaptation method using P-Tuning and a Perceiver to enhance Chinese disordered speech recognition with Whisper, achieving up to 13.04% CER reduction.

Contribution

It introduces Perceiver-Prompt, combining P-Tuning and a Perceiver for effective speaker adaptation in disordered speech recognition.

Findings

Up to 13.04% CER reduction on Chinese dysarthric speech dataset.

Perceiver-Prompt improves recognition performance over fine-tuned Whisper.

Demonstrates effectiveness of speaker prompts in disordered speech recognition.

Abstract

Disordered speech recognition profound implications for improving the quality of life for individuals afflicted with, for example, dysarthria. Dysarthric speech recognition encounters challenges including limited data, substantial dissimilarities between dysarthric and non-dysarthric speakers, and significant speaker variations stemming from the disorder. This paper introduces Perceiver-Prompt, a method for speaker adaptation that utilizes P-Tuning on the Whisper large-scale model. We first fine-tune Whisper using LoRA and then integrate a trainable Perceiver to generate fixed-length speaker prompts from variable-length inputs, to improve model recognition of Chinese dysarthric speech. Experimental results from our Chinese dysarthric speech dataset demonstrate consistent improvements in recognition performance with Perceiver-Prompt. Relative reduction up to 13.04% in CER is obtained over the fine-tuned Whisper.