Investigating self-supervised, weakly supervised and fully supervised training approaches for multi-domain automatic speech recognition: a study on Bangladeshi Bangla

TL;DR

This study compares self-supervised, weakly supervised, and fully supervised training methods for multi-domain Bangla speech recognition, highlighting the effectiveness of self-supervised pre-training and the importance of domain selection.

Contribution

It introduces a novel multi-domain Bangla ASR benchmark and evaluates different training approaches, emphasizing the advantages of self-supervised pre-training for robustness.

Findings

Self-supervised wav2vec 2.0 outperforms other methods in multi-domain ASR.

Models trained on SUBAK.KO struggle with spontaneous speech domains.

BanSpeech benchmark will be publicly available for further research.

Abstract

Despite huge improvements in automatic speech recognition (ASR) employing neural networks, ASR systems still suffer from a lack of robustness and generalizability issues due to domain shifting. This is mainly because principal corpus design criteria are often not identified and examined adequately while compiling ASR datasets. In this study, we investigate the robustness of the state-of-the-art transfer learning approaches such as self-supervised wav2vec 2.0 and weakly supervised Whisper as well as fully supervised convolutional neural networks (CNNs) for multi-domain ASR. We also demonstrate the significance of domain selection while building a corpus by assessing these models on a novel multi-domain Bangladeshi Bangla ASR evaluation benchmark - BanSpeech, which contains approximately 6.52 hours of human-annotated speech and 8085 utterances from 13 distinct domains. SUBAK.KO, a mostly read speech corpus for the morphologically rich language Bangla, has been used to train the ASR systems. Experimental evaluation reveals that self-supervised cross-lingual pre-training is the best strategy compared to weak supervision and full supervision to tackle the multi-domain ASR task. Moreover, the ASR models trained on SUBAK.KO face difficulty recognizing speech from domains with mostly spontaneous speech. The BanSpeech will be publicly available to meet the need for a challenging evaluation benchmark for Bangla ASR.