LLM supervised Pre-training for Multimodal Emotion Recognition in Conversations

TL;DR

This paper introduces a multimodal emotion recognition approach that leverages unsupervised speech transcripts, LLM-guided pseudo-labeling, and hierarchical training to improve accuracy on conversational datasets.

Contribution

It proposes a novel hierarchical training method combining speech and text embeddings with LLM-guided pseudo-labeling for emotion recognition.

Findings

Achieves state-of-the-art results on IEMOCAP and MELD datasets.

Improves emotion recognition accuracy over existing benchmarks.

Effectively integrates speech and text modalities for conversational emotion analysis.

Abstract

Emotion recognition in conversations (ERC) is challenging due to the multimodal nature of the emotion expression. In this paper, we propose to pretrain a text-based recognition model from unsupervised speech transcripts with LLM guidance. These transcriptions are obtained from a raw speech dataset with a pre-trained ASR system. A text LLM model is queried to provide pseudo-labels for these transcripts, and these pseudo-labeled transcripts are subsequently used for learning an utterance level text-based emotion recognition model. We use the utterance level text embeddings for emotion recognition in conversations along with speech embeddings obtained from a recently proposed pre-trained model. A hierarchical way of training the speech-text model is proposed, keeping in mind the conversational nature of the dataset. We perform experiments on three established datasets, namely, IEMOCAP, MELD, and CMU- MOSI, where we illustrate that the proposed model improves over other benchmarks and achieves state-of-the-art results on two out of these three datasets.