"We care": Improving Code Mixed Speech Emotion Recognition in Customer-Care Conversations

TL;DR

This paper introduces NSED, a natural code-mixed speech emotion dataset for customer-care conversations, and demonstrates that incorporating word-level VAD values improves emotion recognition accuracy, especially for negative emotions.

Contribution

The creation of NSED, a natural, code-mixed speech emotion dataset with VAD annotations, and showing that word-level VAD integration enhances SER performance in noisy, real-world scenarios.

Findings

Incorporating word-level VAD improves negative emotion recognition by 2%.

NSED dataset captures natural code-mixed speech in customer care contexts.

Enhanced SER accuracy can lead to more empathetic conversational agents.

Abstract

Speech Emotion Recognition (SER) is the task of identifying the emotion expressed in a spoken utterance. Emotion recognition is essential in building robust conversational agents in domains such as law, healthcare, education, and customer support. Most of the studies published on SER use datasets created by employing professional actors in a noise-free environment. In natural settings such as a customer care conversation, the audio is often noisy with speakers regularly switching between different languages as they see fit. We have worked in collaboration with a leading unicorn in the Conversational AI sector to develop Natural Speech Emotion Dataset (NSED). NSED is a natural code-mixed speech emotion dataset where each utterance in a conversation is annotated with emotion, sentiment, valence, arousal, and dominance (VAD) values. In this paper, we show that by incorporating word-level VAD value we improve on the task of SER by 2%, for negative emotions, over the baseline value for NSED. High accuracy for negative emotion recognition is essential because customers expressing negative opinions/views need to be pacified with urgency, lest complaints and dissatisfaction snowball and get out of hand. Escalation of negative opinions speedily is crucial for business interests. Our study then can be utilized to develop conversational agents which are more polite and empathetic in such situations.