Exploiting Cross Domain Acoustic-to-articulatory Inverted Features For Disordered Speech Recognition

TL;DR

This paper introduces a cross-domain acoustic-to-articulatory inversion method using neural networks to generate articulatory features, significantly improving disordered speech recognition accuracy across datasets.

Contribution

It presents a novel cross-domain A2A inversion approach with neural models and feature adaptation, enhancing disordered speech recognition performance.

Findings

Achieved a WER of 24.82% on UASpeech, the lowest reported.

Incorporating articulatory features outperforms acoustic-only systems.

Multi-modal system with video and data augmentation further improves results.

Abstract

Articulatory features are inherently invariant to acoustic signal distortion and have been successfully incorporated into automatic speech recognition (ASR) systems for normal speech. Their practical application to disordered speech recognition is often limited by the difficulty in collecting such specialist data from impaired speakers. This paper presents a cross-domain acoustic-to-articulatory (A2A) inversion approach that utilizes the parallel acoustic-articulatory data of the 15-hour TORGO corpus in model training before being cross-domain adapted to the 102.7-hour UASpeech corpus and to produce articulatory features. Mixture density networks based neural A2A inversion models were used. A cross-domain feature adaptation network was also used to reduce the acoustic mismatch between the TORGO and UASpeech data. On both tasks, incorporating the A2A generated articulatory features consistently outperformed the baseline hybrid DNN/TDNN, CTC and Conformer based end-to-end systems constructed using acoustic features only. The best multi-modal system incorporating video modality and the cross-domain articulatory features as well as data augmentation and learning hidden unit contributions (LHUC) speaker adaptation produced the lowest published word error rate (WER) of 24.82% on the 16 dysarthric speakers of the benchmark UASpeech task.