Next Tokens Denoising for Speech Synthesis

TL;DR

The paper introduces Dragon-FM, a novel TTS model combining autoregressive and flow-matching techniques to enable fast, coherent, and high-quality speech synthesis, especially suited for long-form content like podcasts.

Contribution

Dragon-FM unifies autoregressive and flow-matching methods for speech synthesis, enabling efficient chunk-wise processing with global coherence and bidirectional context utilization.

Findings

Efficient processing of 48 kHz audio at 12.5 tokens/sec.

High-quality zero-shot podcast generation demonstrated.

Effective integration of continuous and discrete feature modeling.

Abstract

While diffusion and autoregressive (AR) models have significantly advanced generative modeling, they each present distinct limitations. AR models, which rely on causal attention, cannot exploit future context and suffer from slow generation speeds. Conversely, diffusion models struggle with key-value (KV) caching. To overcome these challenges, we introduce Dragon-FM, a novel text-to-speech (TTS) design that unifies AR and flow-matching. This model processes 48 kHz audio codec tokens in chunks at a compact rate of 12.5 tokens per second. This design enables AR modeling across chunks, ensuring global coherence, while parallel flow-matching within chunks facilitates fast iterative denoising. Thus, the model leverages KV-cache across chunks and utilizes bidirectional context within each chunk. Furthermore, it bridges continuous and discrete feature modeling, demonstrating that continuous AR flow-matching can predict discrete tokens with finite scalar quantizers. This efficient codec and fast chunk-autoregressive architecture also make the model highly effective for generating long-form content, such as podcasts. Experiments on podcast datasets demonstrate its capability to efficiently generate high-quality zero-shot podcasts.