Personalized Adaptation with Pre-trained Speech Encoders for Continuous Emotion Recognition

TL;DR

This paper introduces a novel unsupervised personalized speech emotion recognition method that leverages pre-trained speech encoders with speaker embeddings and label distribution compensation, achieving state-of-the-art results.

Contribution

It proposes a new unsupervised approach combining speaker-conditioned pre-training and label shift compensation for improved emotion recognition.

Findings

Outperforms existing personalization baselines.

Achieves state-of-the-art valence estimation performance.

Demonstrates robustness across diverse speakers.

Abstract

There are individual differences in expressive behaviors driven by cultural norms and personality. This between-person variation can result in reduced emotion recognition performance. Therefore, personalization is an important step in improving the generalization and robustness of speech emotion recognition. In this paper, to achieve unsupervised personalized emotion recognition, we first pre-train an encoder with learnable speaker embeddings in a self-supervised manner to learn robust speech representations conditioned on speakers. Second, we propose an unsupervised method to compensate for the label distribution shifts by finding similar speakers and leveraging their label distributions from the training set. Extensive experimental results on the MSP-Podcast corpus indicate that our method consistently outperforms strong personalization baselines and achieves state-of-the-art performance for valence estimation.