SemaVoice: Semantic-Aware Continuous Autoregressive Speech Synthesis

TL;DR

SemaVoice is a novel semantic-aware autoregressive TTS framework that improves zero-shot speech synthesis quality by aligning speech representations with semantic and structural cues.

Contribution

It introduces a Speech Foundation Model guided alignment mechanism to enhance continuous speech representations for high-fidelity zero-shot TTS.

Findings

Achieves 1.71% WER on Seed-TTS benchmark.

Outperforms existing open-source systems in objective and subjective evaluations.

Shows significant improvements with SFM guided alignment across different granularities.

Abstract

Continuous autoregressive speech synthesis has recently emerged as a promising direction for zero-shot text-to-speech (TTS). However, existing methods still suffer from a fundamental mismatch between semantic-prosodic modeling and reconstruction-driven continuous speech representations. This mismatch causes TTS models to focus excessively on low-level acoustic textures at the expense of high-level semantic coherence, further exacerbating error accumulation in autoregressive generation. To address this challenge, we propose SemaVoice, a semantic-aware continuous autoregressive framework for high-fidelity zero-shot TTS. SemaVoice introduces a Speech Foundation Model (SFM) guided alignment mechanism that refines continuous speech representations to better capture both local semantic consistency and global structural relationships. These representations condition a patch-wise diffusion head within the autoregressive framework for high-quality speech synthesis. Experimental results on the Seed-TTS benchmark show that SemaVoice achieves an English WER of 1.71\% and remains highly competitive with state-of-the-art open-source systems in both objective and subjective evaluations. The effectiveness of SFM guided alignment is further confirmed by significant improvements under varying representation granularities with a fixed information-rate constraint.