Towards Unsupervised Speech Recognition and Synthesis with Quantized Speech Representation Learning

TL;DR

This paper introduces SeqRQ-AE, a novel autoencoder that learns phoneme-like speech representations from mostly unpaired audio data, enabling unsupervised speech recognition and synthesis with minimal annotated data.

Contribution

The paper presents a new autoencoder model that learns phoneme-synchronized representations from unpaired audio, requiring only small amounts of annotated data for mapping.

Findings

Learned representations align with IPA vowel chart.

Outperforms existing methods with less than 20 minutes of annotated speech.

Synthesizes intelligible speech surpassing baseline models.

Abstract

In this paper we propose a Sequential Representation Quantization AutoEncoder (SeqRQ-AE) to learn from primarily unpaired audio data and produce sequences of representations very close to phoneme sequences of speech utterances. This is achieved by proper temporal segmentation to make the representations phoneme-synchronized, and proper phonetic clustering to have total number of distinct representations close to the number of phonemes. Mapping between the distinct representations and phonemes is learned from a small amount of annotated paired data. Preliminary experiments on LJSpeech demonstrated the learned representations for vowels have relative locations in latent space in good parallel to that shown in the IPA vowel chart defined by linguistics experts. With less than 20 minutes of annotated speech, our method outperformed existing methods on phoneme recognition and is able to synthesize intelligible speech that beats our baseline model.