Domain-adversarial training of multi-speaker TTS

TL;DR

This paper proposes a domain-adversarial training method for multi-speaker TTS that disentangles speaker characteristics from linguistic content, improving voice similarity and naturalness in synthesized speech.

Contribution

It introduces a gradient reversal layer with an angular margin softmax speaker classifier to unlearn speaker traits from text embeddings in multi-speaker TTS.

Findings

39.9% improvement in similarity MOS

40.1% improvement in naturalness MOS

Effective disentanglement of speaker and linguistic features

Abstract

Multi-speaker TTS has to learn both linguistic embedding and text embedding to generate speech of desired linguistic content in desired voice. However, it is unclear which characteristic of speech results from speaker and which part from linguistic content. In this paper, text embedding is forced to unlearn speaker dependent characteristic using gradient reversal layer to auxiliary speaker classifier that we introduce. We train a speaker classifier using angular margin softmax loss. In subjective evaluation, it is shown that the adversarial training of text embedding for unilingual multi-speaker TTS results in 39.9% improvement on similarity MOS and 40.1% improvement on naturalness MOS.