Improved End-to-End Dysarthric Speech Recognition via Meta-learning Based Model Re-initialization

TL;DR

This paper introduces a meta-learning based re-initialization method for end-to-end dysarthric speech recognition models, significantly improving adaptation and recognition accuracy for limited and acoustically deviant dysarthric speech data.

Contribution

It extends MAML and Reptile algorithms to better initialize models for dysarthric speech recognition, enabling faster adaptation and improved performance.

Findings

Achieved 54.2% relative WER reduction over the base model.

Outperformed direct fine-tuning methods.

Comparable to state-of-the-art hybrid models.

Abstract

Dysarthric speech recognition is a challenging task as dysarthric data is limited and its acoustics deviate significantly from normal speech. Model-based speaker adaptation is a promising method by using the limited dysarthric speech to fine-tune a base model that has been pre-trained from large amounts of normal speech to obtain speaker-dependent models. However, statistic distribution mismatches between the normal and dysarthric speech data limit the adaptation performance of the base model. To address this problem, we propose to re-initialize the base model via meta-learning to obtain a better model initialization. Specifically, we focus on end-to-end models and extend the model-agnostic meta learning (MAML) and Reptile algorithms to meta update the base model by repeatedly simulating adaptation to different dysarthric speakers. As a result, the re-initialized model acquires dysarthric speech knowledge and learns how to perform fast adaptation to unseen dysarthric speakers with improved performance. Experimental results on UASpeech dataset show that the best model with proposed methods achieves 54.2% and 7.6% relative word error rate reduction compared with the base model without finetuning and the model directly fine-tuned from the base model, respectively, and it is comparable with the state-of-the-art hybrid DNN-HMM model.