Wav2vec-C: A Self-supervised Model for Speech Representation Learning

TL;DR

Wav2vec-C presents a self-supervised speech representation learning model that combines contrastive learning with quantization and reconstruction, achieving significant error reduction on speech recognition tasks.

Contribution

It introduces a novel combination of wav2vec 2.0 and VQ-VAE techniques with a regularized quantization process for improved speech representations.

Findings

Achieves twice the error reduction over baseline models.

Utilizes 10k hours of unlabeled data for training.

Improves codebook utilization compared to wav2vec 2.0.

Abstract

Wav2vec-C introduces a novel representation learning technique combining elements from wav2vec 2.0 and VQ-VAE. Our model learns to reproduce quantized representations from partially masked speech encoding using a contrastive loss in a way similar to Wav2vec 2.0. However, the quantization process is regularized by an additional consistency network that learns to reconstruct the input features to the wav2vec 2.0 network from the quantized representations in a way similar to a VQ-VAE model. The proposed self-supervised model is trained on 10k hours of unlabeled data and subsequently used as the speech encoder in a RNN-T ASR model and fine-tuned with 1k hours of labeled data. This work is one of only a few studies of self-supervised learning on speech tasks with a large volume of real far-field labeled data. The Wav2vec-C encoded representations achieves, on average, twice the error reduction over baseline and a higher codebook utilization in comparison to wav2vec 2.0