PFluxTTS: Hybrid Flow-Matching TTS with Robust Cross-Lingual Voice Cloning and Inference-Time Model Fusion

TL;DR

PFluxTTS is a hybrid TTS system that improves cross-lingual voice cloning, naturalness, and audio quality through innovative model fusion and super-resolution vocoding, outperforming existing models.

Contribution

It introduces a dual-decoder design with inference-time fusion, robust cross-lingual cloning with speech prompts, and a super-resolution vocoder, advancing TTS capabilities without extra training.

Findings

Outperforms F5-TTS, FishSpeech, and SparkTTS on cross-lingual data.

Achieves MOS 4.11, 23% lower WER than competitors.

Surpasses ElevenLabs in speaker similarity (+0.32 SMOS).

Abstract

We present PFluxTTS, a hybrid text-to-speech system addressing three gaps in flow-matching TTS: the stability-naturalness trade-off, weak cross-lingual voice cloning, and limited audio quality from low-rate mel features. Our contributions are: (1) a dual-decoder design combining duration-guided and alignment-free models through inference-time vector-field fusion; (2) robust cloning using a sequence of speech-prompt embeddings in a FLUX-based decoder, preserving speaker traits across languages without prompt transcripts; and (3) a modified PeriodWave vocoder with super-resolution to 48 kHz. On cross-lingual in-the-wild data, PFluxTTS clearly outperforms F5-TTS, FishSpeech, and SparkTTS, matches ChatterBox in naturalness (MOS 4.11) while achieving 23% lower WER (6.9% vs. 9.0%), and surpasses ElevenLabs in speaker similarity (+0.32 SMOS). The system remains robust in challenging scenarios where most open-source models fail, while requiring only short reference audio and no extra training. Audio demos are available at https://braskai.github.io/pfluxtts/