Censer: Curriculum Semi-supervised Learning for Speech Recognition Based on Self-supervised Pre-training

TL;DR

Censer introduces a semi-supervised speech recognition method that effectively utilizes unlabeled data through progressive pseudo-labeling, self-supervised pre-training, and novel data management techniques, outperforming existing methods on standard datasets.

Contribution

The paper presents a new semi-supervised learning algorithm for speech recognition that combines self-supervised pre-training with progressive pseudo-labeling and data management strategies.

Findings

Achieves superior performance on Libri-Light and LibriSpeech datasets.

Effectively leverages unlabeled data with pseudo-label quality assessment.

Outperforms existing semi-supervised speech recognition approaches.

Abstract

Recent studies have shown that the benefits provided by self-supervised pre-training and self-training (pseudo-labeling) are complementary. Semi-supervised fine-tuning strategies under the pre-training framework, however, remain insufficiently studied. Besides, modern semi-supervised speech recognition algorithms either treat unlabeled data indiscriminately or filter out noisy samples with a confidence threshold. The dissimilarities among different unlabeled data are often ignored. In this paper, we propose Censer, a semi-supervised speech recognition algorithm based on self-supervised pre-training to maximize the utilization of unlabeled data. The pre-training stage of Censer adopts wav2vec2.0 and the fine-tuning stage employs an improved semi-supervised learning algorithm from slimIPL, which leverages unlabeled data progressively according to their pseudo labels' qualities. We also incorporate a temporal pseudo label pool and an exponential moving average to control the pseudo labels' update frequency and to avoid model divergence. Experimental results on Libri-Light and LibriSpeech datasets manifest our proposed method achieves better performance compared to existing approaches while being more unified.