SingingSDS: A Singing-Capable Spoken Dialogue System for Conversational Roleplay Applications

TL;DR

SingingSDS introduces a novel spoken dialogue system that responds with singing instead of speech, enhancing engagement and emotional impact in character-based roleplay and entertainment scenarios.

Contribution

It presents a modular, open-source framework for singing-based dialogue responses, integrating ASR, LLM, and SVS technologies for customizable, high-quality musical interactions.

Findings

Supports diverse character personas and musical styles

Offers a flexible, plug-and-play web demo

Enables expressive, memorable interactions

Abstract

With recent advances in automatic speech recognition (ASR), large language models (LLMs), and text-to-speech (TTS) technologies, spoken dialogue systems (SDS) have become widely accessible. However, most existing SDS are limited to conventional spoken responses. We present SingingSDS, a cascaded SDS that responds through singing rather than speaking, fostering more affective, memorable, and pleasurable interactions in character-based roleplay and interactive entertainment scenarios. SingingSDS employs a modular ASR-LLM-SVS pipeline and supports a wide range of configurations across character personas, ASR and LLM backends, SVS models, melody sources, and voice profiles, tailored to different needs in terms of latency, quality, and musical style. SingingSDS is available as a plug-and-play web demo, featuring modular, open-source code that supports customization and extension. Demo: https://huggingface.co/spaces/espnet/SingingSDS. Code: https://github.com/SingingSDS/SingingSDS.