Is Style All You Need? Dependencies Between Emotion and GST-based Speaker Recognition

TL;DR

This paper explores using speaker recognition embeddings derived from GST-based models to detect and classify emotions in speech, demonstrating competitive accuracy across multiple datasets.

Contribution

It introduces a novel approach of reusing speaker recognition model weights for emotion classification and proposes a hierarchical classifier to improve accuracy.

Findings

Achieved up to 81.2% accuracy on IEMOCAP

Proposed hierarchical classifier improves accuracy by 2% on CREMA-D

Speaker embeddings effectively encode emotional information

Abstract

In this work, we study the hypothesis that speaker identity embeddings extracted from speech samples may be used for detection and classification of emotion. In particular, we show that emotions can be effectively identified by learning speaker identities by use of a 1-D Triplet Convolutional Neural Network (CNN) & Global Style Token (GST) scheme (e.g., DeepTalk Network) and reusing the trained speaker recognition model weights to generate features in the emotion classification domain. The automatic speaker recognition (ASR) network is trained with VoxCeleb1, VoxCeleb2, and Librispeech datasets with a triplet training loss function using speaker identity labels. Using an Support Vector Machine (SVM) classifier, we map speaker identity embeddings into discrete emotion categories from the CREMA-D, IEMOCAP, and MSP-Podcast datasets. On the task of speech emotion detection, we obtain 80.8% ACC with acted emotion samples from CREMA-D, 81.2% ACC with semi-natural emotion samples in IEMOCAP, and 66.9% ACC with natural emotion samples in MSP-Podcast. We also propose a novel two-stage hierarchical classifier (HC) approach which demonstrates +2% ACC improvement on CREMA-D emotion samples. Through this work, we seek to convey the importance of holistically modeling intra-user variation within audio samples