# Speech Emotion Recognition via Entropy-Aware Score Selection

## TL;DR

This paper introduces a multimodal speech emotion recognition framework that uses entropy-aware score selection to effectively combine speech and text predictions, improving accuracy over single-modality systems.

## Contribution

It presents a novel entropy-based late score fusion method and a sentiment-to-emotion mapping strategy for enhanced multimodal emotion recognition.

## Key findings

- Improved accuracy on IEMOCAP dataset
- Effective multimodal integration using entropy thresholds
- Outperforms traditional single-modality systems

## Abstract

In this paper, we propose a multimodal framework for speech emotion recognition that leverages entropy-aware score selection to combine speech and textual predictions. The proposed method integrates a primary pipeline that consists of an acoustic model based on wav2vec2.0 and a secondary pipeline that consists of a sentiment analysis model using RoBERTa-XLM, with transcriptions generated via Whisper-large-v3. We propose a late score fusion approach based on entropy and varentropy thresholds to overcome the confidence constraints of primary pipeline predictions. A sentiment mapping strategy translates three sentiment categories into four target emotion classes, enabling coherent integration of multimodal predictions. The results on the IEMOCAP and MSP-IMPROV datasets show that the proposed method offers a practical and reliable enhancement over traditional single-modality systems.