Effectiveness of self-supervised pre-training for speech recognition

TL;DR

This paper evaluates self-supervised pre-training methods for speech recognition, showing that quantization-based algorithms like vq-wav2vec improve accuracy and enable effective models with minimal labeled data.

Contribution

It introduces a direct fine-tuning approach of BERT models on transcribed speech and compares raw audio versus spectral features for pre-training.

Findings

vq-wav2vec improves accuracy over non-quantized methods

Near-zero transcribed data can achieve competitive speech recognition performance

Significant WER reduction with minimal labeled data

Abstract

We compare self-supervised representation learning algorithms which either explicitly quantize the audio data or learn representations without quantization. We find the former to be more accurate since it builds a good vocabulary of the data through vq-wav2vec [1] to enable learning of effective representations in subsequent BERT training. Different to previous work, we directly fine-tune the pre-trained BERT models on transcribed speech using a Connectionist Temporal Classification (CTC) loss instead of feeding the representations into a task-specific model. We also propose a BERT-style model learning directly from the continuous audio data and compare pre-training on raw audio to spectral features. Fine-tuning a BERT model on 10 hour of labeled Librispeech data with a vq-wav2vec vocabulary is almost as good as the best known reported system trained on 100 hours of labeled data on testclean, while achieving a 25% WER reduction on test-other. When using only 10 minutes of labeled data, WER is 25.2 on test-other and 16.3 on test-clean. This demonstrates that self-supervision can enable speech recognition systems trained on a near-zero amount of transcribed data.