End-to-End Continuous Speech Emotion Recognition in Real-life Customer Service Call Center Conversations

TL;DR

This paper introduces a large-scale dataset and an end-to-end system for continuous speech emotion recognition in real-life customer service calls, emphasizing contextual factors and multitask learning to improve accuracy.

Contribution

The study presents a new dataset (CusEmo) and an end-to-end SER approach that incorporates contextual information and empathy levels, advancing real-world emotion recognition.

Findings

Incorporating empathy level improves model performance.

Contextual information enhances emotion recognition accuracy.

Large-scale dataset enables robust real-life SER modeling.

Abstract

Speech Emotion recognition (SER) in call center conversations has emerged as a valuable tool for assessing the quality of interactions between clients and agents. In contrast to controlled laboratory environments, real-life conversations take place under uncontrolled conditions and are subject to contextual factors that influence the expression of emotions. In this paper, we present our approach to constructing a large-scale reallife dataset (CusEmo) for continuous SER in customer service call center conversations. We adopted the dimensional emotion annotation approach to capture the subtlety, complexity, and continuity of emotions in real-life call center conversations, while annotating contextual information. The study also addresses the challenges encountered during the application of the End-to-End (E2E) SER system to the dataset, including determining the appropriate label sampling rate and input segment length, as well as integrating contextual information (interlocutor's gender and empathy level) with different weights using multitask learning. The result shows that incorporating the empathy level information improved the model's performance.