Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition

TL;DR

This paper advances semi-supervised learning for automatic speech recognition by combining recent techniques and large datasets to achieve state-of-the-art word-error-rates on LibriSpeech.

Contribution

It introduces a novel combination of noisy student training, SpecAugment, and large Conformer models pre-trained with wav2vec 2.0, pushing the performance limits.

Findings

Achieved WERs of 1.4%/2.6% on LibriSpeech test sets.

Surpassed previous state-of-the-art WERs with new methods.

Demonstrated the effectiveness of large-scale semi-supervised training.

Abstract

We employ a combination of recent developments in semi-supervised learning for automatic speech recognition to obtain state-of-the-art results on LibriSpeech utilizing the unlabeled audio of the Libri-Light dataset. More precisely, we carry out noisy student training with SpecAugment using giant Conformer models pre-trained using wav2vec 2.0 pre-training. By doing so, we are able to achieve word-error-rates (WERs) 1.4%/2.6% on the LibriSpeech test/test-other sets against the current state-of-the-art WERs 1.7%/3.3%.