EmoNews: A Spoken Dialogue System for Expressive News Conversations

TL;DR

This paper presents EmoNews, a task-oriented spoken dialogue system that uses emotional speech regulation based on context to enhance empathetic news conversations, leveraging large language models and emotional TTS.

Contribution

It introduces a novel emotional SDS for news that combines LLM-based sentiment analysis with PromptTTS for context-aware emotional speech synthesis, and proposes a new evaluation scale.

Findings

The emotional SDS outperforms baseline in emotion regulation.

The system increases user engagement in news conversations.

Open-source code is provided for reproducibility.

Abstract

We develop a task-oriented spoken dialogue system (SDS) that regulates emotional speech based on contextual cues to enable more empathetic news conversations. Despite advancements in emotional text-to-speech (TTS) techniques, task-oriented emotional SDSs remain underexplored due to the compartmentalized nature of SDS and emotional TTS research, as well as the lack of standardized evaluation metrics for social goals. We address these challenges by developing an emotional SDS for news conversations that utilizes a large language model (LLM)-based sentiment analyzer to identify appropriate emotions and PromptTTS to synthesize context-appropriate emotional speech. We also propose subjective evaluation scale for emotional SDSs and judge the emotion regulation performance of the proposed and baseline systems. Experiments showed that our emotional SDS outperformed a baseline system in terms of the emotion regulation and engagement. These results suggest the critical role of speech emotion for more engaging conversations. All our source code is open-sourced at https://github.com/dhatchi711/espnet-emotional-news/tree/emo-sds/egs2/emo_news_sds/sds1