WavThruVec: Latent speech representation as intermediate features for neural speech synthesis

TL;DR

WavThruVec introduces a two-stage neural speech synthesis model using high-level Wav2Vec 2.0 embeddings as intermediate features, enhancing robustness, generalization, and enabling zero-shot synthesis.

Contribution

The paper proposes WavThruVec, a novel two-stage architecture leveraging Wav2Vec 2.0 embeddings for improved neural speech synthesis, addressing limitations of traditional intermediate features.

Findings

Matches state-of-the-art synthesis quality

Improves robustness to noise and out-of-vocabulary words

Enables zero-shot voice synthesis and conversion

Abstract

Recent advances in neural text-to-speech research have been dominated by two-stage pipelines utilizing low-level intermediate speech representation such as mel-spectrograms. However, such predetermined features are fundamentally limited, because they do not allow to exploit the full potential of a data-driven approach through learning hidden representations. For this reason, several end-to-end methods have been proposed. However, such models are harder to train and require a large number of high-quality recordings with transcriptions. Here, we propose WavThruVec - a two-stage architecture that resolves the bottleneck by using high-dimensional Wav2Vec 2.0 embeddings as intermediate speech representation. Since these hidden activations provide high-level linguistic features, they are more robust to noise. That allows us to utilize annotated speech datasets of a lower quality to train the first-stage module. At the same time, the second-stage component can be trained on large-scale untranscribed audio corpora, as Wav2Vec 2.0 embeddings are already time-aligned. This results in an increased generalization capability to out-of-vocabulary words, as well as to a better generalization to unseen speakers. We show that the proposed model not only matches the quality of state-of-the-art neural models, but also presents useful properties enabling tasks like voice conversion or zero-shot synthesis.