BreezyVoice: Adapting TTS for Taiwanese Mandarin with Enhanced Polyphone Disambiguation -- Challenges and Insights

TL;DR

BreezyVoice is a TTS system tailored for Taiwanese Mandarin that effectively handles polyphone disambiguation and generates high-fidelity speech, demonstrating robustness across various contexts and speaker variations.

Contribution

The paper introduces BreezyVoice, integrating novel components like the $S^{3}$ tokenizer and OT-CFM to improve polyphone disambiguation and speech realism in Taiwanese Mandarin TTS.

Findings

Superior performance in general and code-switching contexts

Enhanced robustness for long-tail speakers

Valuable insights into neural codec TTS systems

Abstract

We present BreezyVoice, a Text-to-Speech (TTS) system specifically adapted for Taiwanese Mandarin, highlighting phonetic control abilities to address the unique challenges of polyphone disambiguation in the language. Building upon CosyVoice, we incorporate a $S^{3}$ tokenizer, a large language model (LLM), an optimal-transport conditional flow matching model (OT-CFM), and a grapheme to phoneme prediction model, to generate realistic speech that closely mimics human utterances. Our evaluation demonstrates BreezyVoice's superior performance in both general and code-switching contexts, highlighting its robustness and effectiveness in generating high-fidelity speech. Additionally, we address the challenges of generalizability in modeling long-tail speakers and polyphone disambiguation. Our approach significantly enhances performance and offers valuable insights into the workings of neural codec TTS systems.