Disentangling Style and Speaker Attributes for TTS Style Transfer

TL;DR

This paper introduces a novel neural TTS method that effectively transfers speech styles, including unseen styles, by disentangling style and speaker attributes using advanced inference and embedding techniques, outperforming prior systems.

Contribution

The paper proposes a new approach for seen and unseen style transfer in neural TTS using IAF and a speaker encoder, trained with six objectives for improved robustness and effectiveness.

Findings

Superior performance in style transfer tasks compared to prior methods

Effective handling of unseen styles with robustness

Enhanced style and speaker disentanglement

Abstract

End-to-end neural TTS has shown improved performance in speech style transfer. However, the improvement is still limited by the available training data in both target styles and speakers. Additionally, degenerated performance is observed when the trained TTS tries to transfer the speech to a target style from a new speaker with an unknown, arbitrary style. In this paper, we propose a new approach to seen and unseen style transfer training on disjoint, multi-style datasets, i.e., datasets of different styles are recorded, one individual style by one speaker in multiple utterances. An inverse autoregressive flow (IAF) technique is first introduced to improve the variational inference for learning an expressive style representation. A speaker encoder network is then developed for learning a discriminative speaker embedding, which is jointly trained with the rest neural TTS modules. The proposed approach of seen and unseen style transfer is effectively trained with six specifically-designed objectives: reconstruction loss, adversarial loss, style distortion loss, cycle consistency loss, style classification loss, and speaker classification loss. Experiments demonstrate, both objectively and subjectively, the effectiveness of the proposed approach for seen and unseen style transfer tasks. The performance of our approach is superior to and more robust than those of four other reference systems of prior art.