Leveraging Self-Supervised Models for Automatic Whispered Speech Recognition

TL;DR

This paper introduces a novel self-supervised WavLM-based approach for recognizing Irish dialect whispered speech, significantly outperforming existing models like OpenAI Whisper in accuracy.

Contribution

It presents a fine-tuned WavLM model specifically adapted for whispered speech recognition in Irish dialects, addressing the scarcity of training data and acoustic challenges.

Findings

WavLM-based system achieved WER of 9.22%

Significant improvement over OpenAI Whisper baseline

Demonstrates effectiveness for dialect-specific whispered speech

Abstract

In automatic speech recognition, any factor that alters the acoustic properties of speech can pose a challenge to the system's performance. This paper presents a novel approach for automatic whispered speech recognition in the Irish dialect using the self-supervised WavLM model. Conventional automatic speech recognition systems often fail to accurately recognise whispered speech due to its distinct acoustic properties and the scarcity of relevant training data. To address this challenge, we utilized a pre-trained WavLM model, fine-tuned with a combination of whispered and normal speech data from the wTIMIT and CHAINS datasets, which include the English language in Singaporean and Irish dialects, respectively. Our baseline evaluation with the OpenAI Whisper model highlighted its limitations, achieving a Word Error Rate (WER) of 18.8% and a Character Error Rate (CER) of 4.24% on whispered speech. In contrast, the proposed WavLM-based system significantly improved performance, achieving a WER of 9.22% and a CER of 2.59%. These results demonstrate the efficacy of our approach in recognising whispered speech and underscore the importance of tailored acoustic modeling for robust automatic speech recognition systems. This study provides valuable insights into developing effective automatic speech recognition solutions for challenging speech affected by whisper and dialect. The source codes for this paper are freely available.