vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations

TL;DR

The paper introduces vq-wav2vec, a self-supervised method for learning discrete speech representations using quantization techniques, improving phoneme classification and speech recognition performance.

Contribution

It presents a novel approach combining self-supervised learning with discrete representation learning for speech, enabling NLP algorithms to be applied directly to audio.

Findings

Achieved state-of-the-art results on TIMIT phoneme classification.

Improved WSJ speech recognition performance.

Demonstrated effectiveness of discretization in speech tasks.

Abstract

We propose vq-wav2vec to learn discrete representations of audio segments through a wav2vec-style self-supervised context prediction task. The algorithm uses either a gumbel softmax or online k-means clustering to quantize the dense representations. Discretization enables the direct application of algorithms from the NLP community which require discrete inputs. Experiments show that BERT pre-training achieves a new state of the art on TIMIT phoneme classification and WSJ speech recognition.