Continuous Metric Learning For Transferable Speech Emotion Recognition and Embedding Across Low-resource Languages

TL;DR

This paper introduces a novel continuous metric learning approach for speech emotion recognition that enhances transferability across low-resource languages by leveraging semi-supervised learning and dimensional emotion annotations.

Contribution

It proposes the first continuous metric learning method for SER, improving transferability and performance in low-resource language scenarios through semi-supervised autoencoder training.

Findings

Outperforms baseline unsupervised autoencoder in emotion classification accuracy

Achieves better correlation with dimensional emotion variables

Comparable to BERT-based models with lower complexity

Abstract

Speech emotion recognition~(SER) refers to the technique of inferring the emotional state of an individual from speech signals. SERs continue to garner interest due to their wide applicability. Although the domain is mainly founded on signal processing, machine learning, and deep learning, generalizing over languages continues to remain a challenge. However, developing generalizable and transferable models are critical due to a lack of sufficient resources in terms of data and labels for languages beyond the most commonly spoken ones. To improve performance over languages, we propose a denoising autoencoder with semi-supervision using a continuous metric loss based on either activation or valence. The novelty of this work lies in our proposal of continuous metric learning, which is among the first proposals on the topic to the best of our knowledge. Furthermore, to address the lack of activation and valence labels in the transfer datasets, we annotate the signal samples with activation and valence levels corresponding to a dimensional model of emotions, which were then used to evaluate the quality of the embedding over the transfer datasets. We show that the proposed semi-supervised model consistently outperforms the baseline unsupervised method, which is a conventional denoising autoencoder, in terms of emotion classification accuracy as well as correlation with respect to the dimensional variables. Further evaluation of classification accuracy with respect to the reference, a BERT based speech representation model, shows that the proposed method is comparable to the reference method in classifying specific emotion classes at a much lower complexity.