M3-TTS: Multi-modal DiT Alignment & Mel-latent for Zero-shot High-fidelity Speech Synthesis

TL;DR

M3-TTS introduces a multi-modal diffusion transformer-based non-autoregressive speech synthesis model that achieves high-fidelity, natural-sounding speech with efficient training and stable alignment without pseudo-alignments.

Contribution

It proposes a novel multi-modal diffusion transformer architecture for zero-shot high-fidelity TTS, eliminating the need for pseudo-alignment and improving efficiency and naturalness.

Findings

Achieves state-of-the-art NAR TTS performance on benchmarks.

Lowest word error rates of 1.36% (English) and 1.31% (Chinese).

Maintains competitive naturalness scores.

Abstract

Non-autoregressive (NAR) text-to-speech synthesis relies on length alignment between text sequences and audio representations, constraining naturalness and expressiveness. Existing methods depend on duration modeling or pseudo-alignment strategies that severely limit naturalness and computational efficiency. We propose M3-TTS, a concise and efficient NAR TTS paradigm based on multi-modal diffusion transformer (MM-DiT) architecture. M3-TTS employs joint diffusion transformer layers for cross-modal alignment, achieving stable monotonic alignment between variable-length text-speech sequences without pseudo-alignment requirements. Single diffusion transformer layers further enhance acoustic detail modeling. The framework integrates a mel-vae codec that provides 3* training acceleration. Experimental results on Seed-TTS and AISHELL-3 benchmarks demonstrate that M3-TTS achieves state-of-the-art NAR performance with the lowest word error rates (1.36\% English, 1.31\% Chinese) while maintaining competitive naturalness scores. Code and demos will be available at https://wwwwxp.github.io/M3-TTS.