Speaker Conditional WaveRNN: Towards Universal Neural Vocoder for Unseen Speaker and Recording Conditions

TL;DR

This paper introduces SC-WaveRNN, a universal neural vocoder that generalizes well to unseen speakers and recording conditions by incorporating speaker embeddings, significantly improving speech synthesis quality across diverse scenarios.

Contribution

The paper proposes SC-WaveRNN, a novel variant of WaveRNN that effectively handles unseen speakers and recording conditions using speaker embeddings, advancing neural vocoder generalization.

Findings

SC-WaveRNN outperforms baseline WaveRNN on subjective and objective metrics.

Achieves up to 95% MOS improvement for unseen speakers and conditions.

Preferred over baseline TTS by 60% for both seen and unseen speakers.

Abstract

Recent advancements in deep learning led to human-level performance in single-speaker speech synthesis. However, there are still limitations in terms of speech quality when generalizing those systems into multiple-speaker models especially for unseen speakers and unseen recording qualities. For instance, conventional neural vocoders are adjusted to the training speaker and have poor generalization capabilities to unseen speakers. In this work, we propose a variant of WaveRNN, referred to as speaker conditional WaveRNN (SC-WaveRNN). We target towards the development of an efficient universal vocoder even for unseen speakers and recording conditions. In contrast to standard WaveRNN, SC-WaveRNN exploits additional information given in the form of speaker embeddings. Using publicly-available data for training, SC-WaveRNN achieves significantly better performance over baseline WaveRNN on both subjective and objective metrics. In MOS, SC-WaveRNN achieves an improvement of about 23% for seen speaker and seen recording condition and up to 95% for unseen speaker and unseen condition. Finally, we extend our work by implementing a multi-speaker text-to-speech (TTS) synthesis similar to zero-shot speaker adaptation. In terms of performance, our system has been preferred over the baseline TTS system by 60% over 15.5% and by 60.9% over 32.6%, for seen and unseen speakers, respectively.