TADA: A Generative Framework for Speech Modeling via Text-Acoustic Dual Alignment

TL;DR

This paper introduces a novel speech tokenization method that synchronizes text and acoustic features, enabling efficient, high-fidelity speech modeling within large language models, reducing hallucinations and improving performance.

Contribution

The paper proposes a new tokenization scheme for speech that aligns acoustic features with text tokens, allowing unified modeling in LLMs and enhancing speech synthesis and understanding.

Findings

Achieves competitive performance with state-of-the-art TTS and SLM systems.

Virtually eliminates content hallucinations in speech generation.

Reduces inference cost significantly.

Abstract

Modern Text-to-Speech (TTS) systems increasingly leverage Large Language Model (LLM) architectures to achieve scalable, high-fidelity, zero-shot generation. However, these systems typically rely on fixed-frame-rate acoustic tokenization, resulting in speech sequences that are significantly longer than, and asynchronous with their corresponding text. Beyond computational inefficiency, this sequence length disparity often triggers hallucinations in TTS and amplifies the modality gap in spoken language modeling (SLM). In this paper, we propose a novel tokenization scheme that establishes one-to-one synchronization between continuous acoustic features and text tokens, enabling unified, single-stream modeling within an LLM. We demonstrate that these synchronous tokens maintain high-fidelity audio reconstruction and can be effectively modeled in a latent space by a large language model with a flow matching head. Moreover, the ability to seamlessly toggle speech modality within the context enables text-only guidance--a technique that blends logits from text-only and text-speech modes to flexibly bridge the gap toward text-only LLM intelligence. Experimental results indicate that our approach achieves performance competitive with state-of-the-art TTS and SLM systems while virtually eliminating content hallucinations and preserving linguistic integrity, all at a significantly reduced inference cost.