A Study into Pre-training Strategies for Spoken Language Understanding on Dysarthric Speech

TL;DR

This paper explores pre-training strategies for end-to-end spoken language understanding systems tailored to dysarthric speech, emphasizing transfer learning from normal and dysarthric speech to improve performance in low-resource scenarios.

Contribution

It introduces a two-stage pre-training approach for SLU on dysarthric speech and analyzes how impairment severity affects model generalization.

Findings

Pre-training on normal speech followed by fine-tuning improves SLU performance.

Intelligibility scores correlate with model generalization to dysarthric speech.

Pre-training strategies mitigate data scarcity issues in dysarthric SLU.

Abstract

End-to-end (E2E) spoken language understanding (SLU) systems avoid an intermediate textual representation by mapping speech directly into intents with slot values. This approach requires considerable domain-specific training data. In low-resource scenarios this is a major concern, e.g., in the present study dealing with SLU for dysarthric speech. Pretraining part of the SLU model for automatic speech recognition targets helps but no research has shown to which extent SLU on dysarthric speech benefits from knowledge transferred from other dysarthric speech tasks. This paper investigates the efficiency of pre-training strategies for SLU tasks on dysarthric speech. The designed SLU system consists of a TDNN acoustic model for feature encoding and a capsule network for intent and slot decoding. The acoustic model is pre-trained in two stages: initialization with a corpus of normal speech and finetuning on a mixture of dysarthric and normal speech. By introducing the intelligibility score as a metric of the impairment severity, this paper quantitatively analyzes the relation between generalization and pathology severity for dysarthric speech.