Analysis and Utilization of Entrainment on Acoustic and Emotion Features in User-agent Dialogue

TL;DR

This paper investigates how entrainment in acoustic and emotion features affects dialogue systems, demonstrating that acoustic entrainment improves TTS performance but emotion entrainment does not show clear benefits.

Contribution

It provides empirical evidence of entrainment in acoustic and emotion features and explores their integration into TTS systems for enhanced dialogue performance.

Findings

Strong evidence of entrainment in acoustic and emotion features.

Entrainment in acoustic features improves TTS performance.

Emotion feature entrainment does not significantly enhance synthesis.

Abstract

Entrainment is the phenomenon by which an interlocutor adapts their speaking style to align with their partner in conversations. It has been found in different dimensions as acoustic, prosodic, lexical or syntactic. In this work, we explore and utilize the entrainment phenomenon to improve spoken dialogue systems for voice assistants. We first examine the existence of the entrainment phenomenon in human-to-human dialogues in respect to acoustic feature and then extend the analysis to emotion features. The analysis results show strong evidence of entrainment in terms of both acoustic and emotion features. Based on this findings, we implement two entrainment policies and assess if the integration of entrainment principle into a Text-to-Speech (TTS) system improves the synthesis performance and the user experience. It is found that the integration of the entrainment principle into a TTS system brings performance improvement when considering acoustic features, while no obvious improvement is observed when considering emotion features.