ZipVoice-Dialog: Non-Autoregressive Spoken Dialogue Generation with Flow Matching

TL;DR

ZipVoice-Dialog introduces a non-autoregressive flow-matching model for spoken dialogue generation, enhancing speed, stability, and speaker turn accuracy, supported by a new large-scale dataset and evaluation benchmark.

Contribution

The paper presents a novel non-autoregressive flow-matching approach for dialogue generation, along with curriculum learning, speaker-turn embeddings, and a large open dataset for training and evaluation.

Findings

ZipVoice-Dialog achieves faster inference and higher speech intelligibility.

The model demonstrates improved speaker turn-taking accuracy.

The OpenDialog dataset enables comprehensive benchmarking of dialogue models.

Abstract

Generating spoken dialogue is inherently more complex than monologue text-to-speech (TTS), as it demands both realistic turn-taking and the maintenance of distinct speaker timbres. While existing autoregressive (AR) models have made progress, they often suffer from high inference latency and stability issues. To overcome these limitations, we propose ZipVoice-Dialog, a non-autoregressive (NAR) zero-shot spoken dialogue generation model based on flow-matching. Observing that applying vanilla flow-matching to dialogue generation leads to poor speech intelligibility and turn-taking precision, we introduce two simple yet effective methods to adapt flow-matching architectures for dialogue generation: (1) a curriculum learning strategy to ensure robust speech-text alignment, and (2) speaker-turn embeddings to govern precise speaker turn-taking. Additionally, we introduce dedicated strategies to support stereo dialogue generation. Recognizing the lack of training datasets in this field, we curate and release OpenDialog, the first large-scale (6.8k hours) open-source spoken dialogue dataset derived from in-the-wild speech data. Moreover, for fair and rigorous evaluations, we established a benchmark to comprehensively evaluate dialogue generation models. Experiments demonstrate the effectiveness of the proposed methods and dataset, showing that ZipVoice-Dialog achieves superior performance in inference speed, intelligibility, speaker turn-taking accuracy, and speaker similarity. Our code, model checkpoints, and the OpenDialog dataset are publicly available at https://github.com/k2-fsa/ZipVoice.