SLIDE: Integrating Speech Language Model with LLM for Spontaneous Spoken Dialogue Generation

TL;DR

This paper introduces SLIDE, a system that combines large language models and speech language models to generate natural, semantically coherent spontaneous spoken dialogues from text, improving speech realism.

Contribution

The novel integration of LLMs with speech language models for spontaneous dialogue generation, including a phoneme-based conversion and duration prediction approach.

Findings

Generated dialogues are naturalistic and semantically coherent.

System outperforms baseline in speech realism on Fisher dataset.

Effective in maintaining semantic coherence in speech synthesis.

Abstract

Recently, ``textless" speech language models (SLMs) based on speech units have made huge progress in generating naturalistic speech, including non-verbal vocalizations. However, the generated speech samples often lack semantic coherence. In this paper, we propose SLM and LLM Integration for spontaneous spoken Dialogue gEneration (SLIDE). Specifically, we first utilize an LLM to generate the textual content of spoken dialogue. Next, we convert the textual dialogues into phoneme sequences and use a two-tower transformer-based duration predictor to predict the duration of each phoneme. Finally, an SLM conditioned on the spoken phoneme sequences is used to vocalize the textual dialogue. Experimental results on the Fisher dataset demonstrate that our system can generate naturalistic spoken dialogue while maintaining high semantic coherence.