StyleFusion TTS: Multimodal Style-control and Enhanced Feature Fusion for Zero-shot Text-to-speech Synthesis

TL;DR

StyleFusion-TTS is a novel zero-shot TTS system that uses multimodal inputs and hierarchical conformer-based feature fusion to improve style and speaker control, enhancing naturalness and editability.

Contribution

It introduces a general front-end encoder for multimodal inputs and a hierarchical conformer structure for effective feature fusion in zero-shot TTS.

Findings

Promising subjective and objective evaluation results

Effective disentanglement of style and speaker embeddings

Enhanced naturalness and controllability in zero-shot TTS

Abstract

We introduce StyleFusion-TTS, a prompt and/or audio referenced, style and speaker-controllable, zero-shot text-to-speech (TTS) synthesis system designed to enhance the editability and naturalness of current research literature. We propose a general front-end encoder as a compact and effective module to utilize multimodal inputs including text prompts, audio references, and speaker timbre references in a fully zero-shot manner and produce disentangled style and speaker control embeddings. Our novel approach also leverages a hierarchical conformer structure for the fusion of style and speaker control embeddings, aiming to achieve optimal feature fusion within the current advanced TTS architecture. StyleFusion-TTS is evaluated through multiple metrics, both subjectively and objectively. The system shows promising performance across our evaluations, suggesting its potential to contribute to the advancement of the field of zero-shot text-to-speech synthesis.