Beyond Silent Letters: Amplifying LLMs in Emotion Recognition with Vocal Nuances

TL;DR

This paper presents SpeechCueLLM, a novel method that enables Large Language Models to perform multimodal emotion recognition from speech by translating vocal nuances into natural language descriptions, significantly improving accuracy.

Contribution

The paper introduces SpeechCueLLM, a minimal approach that enhances LLM-based emotion recognition by converting speech features into text prompts without architectural modifications.

Findings

Over 2% increase in F1 score on IEMOCAP dataset.

Effective use of speech descriptions improves emotion recognition accuracy.

Method outperforms baseline models requiring structural changes.

Abstract

Emotion recognition in speech is a challenging multimodal task that requires understanding both verbal content and vocal nuances. This paper introduces a novel approach to emotion detection using Large Language Models (LLMs), which have demonstrated exceptional capabilities in natural language understanding. To overcome the inherent limitation of LLMs in processing audio inputs, we propose SpeechCueLLM, a method that translates speech characteristics into natural language descriptions, allowing LLMs to perform multimodal emotion analysis via text prompts without any architectural changes. Our method is minimal yet impactful, outperforming baseline models that require structural modifications. We evaluate SpeechCueLLM on two datasets: IEMOCAP and MELD, showing significant improvements in emotion recognition accuracy, particularly for high-quality audio data. We also explore the effectiveness of various feature representations and fine-tuning strategies for different LLMs. Our experiments demonstrate that incorporating speech descriptions yields a more than 2% increase in the average weighted F1 score on IEMOCAP (from 70.111% to 72.596%).