Chain-of-Thought Training for Open E2E Spoken Dialogue Systems

TL;DR

This paper introduces a chain-of-thought training strategy for end-to-end spoken dialogue systems, improving semantic coherence and training efficiency on limited data by aligning with multimodal language model pre-training.

Contribution

It proposes a novel chain-of-thought formulation that enhances E2E spoken dialogue systems, enabling effective training with limited data and improving response quality.

Findings

Achieved over 1.5 ROUGE-1 improvement over baseline.

Successfully trained on 300 hours of conversation data.

Models and code will be publicly released.

Abstract

Unlike traditional cascaded pipelines, end-to-end (E2E) spoken dialogue systems preserve full differentiability and capture non-phonemic information, making them well-suited for modeling spoken interactions. However, existing E2E approaches often require large-scale training data and generates responses lacking semantic coherence. We propose a simple yet effective strategy leveraging a chain-of-thought (CoT) formulation, ensuring that training on conversational data remains closely aligned with the multimodal language model (LM)'s pre-training on speech recognition~(ASR), text-to-speech synthesis (TTS), and text LM tasks. Our method achieves over 1.5 ROUGE-1 improvement over the baseline, successfully training spoken dialogue systems on publicly available human-human conversation datasets, while being compute-efficient enough to train on just 300 hours of public human-human conversation data, such as the Switchboard. We will publicly release our models and training code.