Towards human-like spoken dialogue generation between AI agents from written dialogue

TL;DR

This paper introduces CHATS, a novel system that converts written dialogues into human-like spoken dialogues by modeling turn-taking, backchannels, and laughter, improving naturalness and fluidity in AI-generated speech.

Contribution

It presents CHATS, a discrete token-based TTS system capable of generating both speaker and listener speech from written dialogue, enhancing natural interaction without needing transcriptions of non-verbal cues.

Findings

CHATS produces more interactive and fluid spoken dialogues than baseline systems.

The system maintains clarity and intelligibility in generated speech.

Experimental results show improved turn-taking and naturalness.

Abstract

The advent of large language models (LLMs) has made it possible to generate natural written dialogues between two agents. However, generating human-like spoken dialogues from these written dialogues remains challenging. Spoken dialogues have several unique characteristics: they frequently include backchannels and laughter, and the smoothness of turn-taking significantly influences the fluidity of conversation. This study proposes CHATS - CHatty Agents Text-to-Speech - a discrete token-based system designed to generate spoken dialogues based on written dialogues. Our system can generate speech for both the speaker side and the listener side simultaneously, using only the transcription from the speaker side, which eliminates the need for transcriptions of backchannels or laughter. Moreover, CHATS facilitates natural turn-taking; it determines the appropriate duration of silence after each utterance in the absence of overlap, and it initiates the generation of overlapping speech based on the phoneme sequence of the next utterance in case of overlap. Experimental evaluations indicate that CHATS outperforms the text-to-speech baseline, producing spoken dialogues that are more interactive and fluid while retaining clarity and intelligibility.