Improving Performance of Seen and Unseen Speech Style Transfer in End-to-end Neural TTS

TL;DR

This paper introduces a novel neural TTS style transfer method capable of handling both seen and unseen styles using disjoint multi-style datasets, leveraging IAF for style encoding and multiple loss functions for improved transfer quality.

Contribution

The paper proposes a new style transfer approach with IAF-based style encoding and a multi-loss training framework, enhancing transfer performance for unseen styles in neural TTS.

Findings

Outperforms baseline systems in style transfer quality.

Effective for both seen and unseen styles.

Improves robustness and naturalness of transferred speech.

Abstract

End-to-end neural TTS training has shown improved performance in speech style transfer. However, the improvement is still limited by the training data in both target styles and speakers. Inadequate style transfer performance occurs when the trained TTS tries to transfer the speech to a target style from a new speaker with an unknown, arbitrary style. In this paper, we propose a new approach to style transfer for both seen and unseen styles, with disjoint, multi-style datasets, i.e., datasets of different styles are recorded, each individual style is by one speaker with multiple utterances. To encode the style information, we adopt an inverse autoregressive flow (IAF) structure to improve the variational inference. The whole system is optimized to minimize a weighed sum of four different loss functions: 1) a reconstruction loss to measure the distortions in both source and target reconstructions; 2) an adversarial loss to "fool" a well-trained discriminator; 3) a style distortion loss to measure the expected style loss after the transfer; 4) a cycle consistency loss to preserve the speaker identity of the source after the transfer. Experiments demonstrate, both objectively and subjectively, the effectiveness of the proposed approach for seen and unseen style transfer tasks. The performance of the new approach is better and more robust than those of four baseline systems of the prior art.