Speech Emotion Recognition via Entropy-Aware Score Selection

TL;DR

This paper introduces a multimodal speech emotion recognition framework that uses entropy-aware score selection to effectively combine speech and text predictions, improving accuracy over single-modality systems.

Contribution

It presents a novel entropy-based late score fusion method and a sentiment-to-emotion mapping strategy for enhanced multimodal emotion recognition.

Findings

Improved accuracy on IEMOCAP dataset

Effective multimodal integration using entropy thresholds

Outperforms traditional single-modality systems

Abstract

In this paper, we propose a multimodal framework for speech emotion recognition that leverages entropy-aware score selection to combine speech and textual predictions. The proposed method integrates a primary pipeline that consists of an acoustic model based on wav2vec2.0 and a secondary pipeline that consists of a sentiment analysis model using RoBERTa-XLM, with transcriptions generated via Whisper-large-v3. We propose a late score fusion approach based on entropy and varentropy thresholds to overcome the confidence constraints of primary pipeline predictions. A sentiment mapping strategy translates three sentiment categories into four target emotion classes, enabling coherent integration of multimodal predictions. The results on the IEMOCAP and MSP-IMPROV datasets show that the proposed method offers a practical and reliable enhancement over traditional single-modality systems.

Tables

Table 1. Table 3: Average results for UA, WA, F1 for varentropy, entropy and entropy + varentropy on the IEMOCAP dataset

Modality	Score Selection	UA (%)	WA (%)	F1 (%)
S	w/o	65.36	64.64	64.01
S+T	Entropy	65.87	63.85	64.46
S+T	Varentropy	65.56	64.81	64.24
S+T	Entropy + Varentropy	65.81	65.05	64.55

Equations

$$\mathcal{H}=\mathcal{H}({\mathbf{p}}^s)=-\sum _{i=1}^N{p}_c^s\log (p_c^s).$$

$$\mathcal{V}=\mathcal{V}({\mathbf{p}}^s)=\sum _{i=1}^N{p}_c^s{\left(\log (p_c^s)+\mathcal{H}\right)}^2.$$

$${\tau }_e$$

$${\tau }_v$$

$${\tau }_{\text{e}}=P_{75}(\mathcal{H})-\Delta +k\delta ,{\tau }_{\text{v}}=P_{25}(\mathcal{V})-\Delta +l\delta ,$$

$$({\tau }_{\text{e}}^c,{\tau }_{\text{v}}^c)=\arg \underset{({\tau }_{\text{e}},{\tau }_{\text{v}})}{\max }\mathcal{M}({\tau }_{\text{e}},{\tau }_{\text{v}}),$$

$$\displaystyle\mathcal{M}(\tau_{\text{e}},\tau_{\text{v}})=\text{Accuracy}=\left(\frac{D}{T}\right)\big{|}_{\tau_{\text{e}},\tau_{\text{v}}}\times 100,$$

Full text

1 Introduction

Speech Emotion Recognition (SER), which aims to recognise emotions directly from voice inputs as discrete emotion classes [1], has become a crucial area of study in human-computer interaction, enhancing the emotional intelligence of virtual assistants, interactive robots, and mental health monitoring systems [2]. The rapid development of deep SER models, such as Convolutional Neural Networks (CNNs) [3], Recurrent Neural Networks (RNNs) [4], and Transformer-based architectures [5, 6, 7], has substantially improved recognition accuracy by capturing complex temporal and contextual patterns in speech. Despite these advances, SER remains challenging due to the subtlety and complexity of emotional expression, limited data availability, and ambiguous labeling, which often lead to misclassification [8, 9].

To address these issues, multimodal approaches that combine speech with textual or visual information have been explored to improve robustness [10, 11]. Among these, integrating speech with text is particularly practical, as textual data can be obtained through automatic speech recognition (ASR) even when authentic transcripts are unavailable [12]. These transcripts are then processed using pretrained Transformer-based text models, such as BERT [13] or RoBERTa [14], to extract rich textual features, while speech models are employed to extract acoustic features.

The key challenge lies in effectively fusing these two modalities. Fusion approaches can be broadly categorized into three types: Early fusion merges raw or low-level features from each modality but often struggles with alignment and dimensionality mismatches [10, 15]. Intermediate fusion learns joint representations from both modalities, offering deeper integration but increasing training complexity [11, 16]. Late fusion, or decision-level fusion, enhances flexibility by allowing each modality to operate independently before merging its outputs, reducing the impact of modality-specific errors [17]. Techniques such as score averaging or rule-based merging further leverage the complementary strengths of different models while supporting independent updates to each component [18].

This work focuses on multimodal emotion recognition and proposes an entropy-aware late score selection strategy. A speech utterance is processed through two branches and obtains two scores. The primary speech branch utilizes a self-supervised learning model as a feature extractor, followed by a classifier that generates emotion predictions covering the full range of emotion classes, four classes in our experiments. The secondary textual branch processes the speech using an ASR model to obtain transcriptions, which are then fed into pretrained sentiment models applied off-the-shelf, without fine-tuning on emotion datasets to produce scores for three sentiment categories: Positive, Neutral, and Negative.

The proposed entropy-aware score selection strategy guides the fusion of speech and sentiment scores. The first step is to evaluate the confidence of the speech score to determine whether intervention from the secondary model is necessary. This decision is based on whether the entropy and varentropy values of the speech score exceed delicately determined thresholds. If the speech score exhibits low confidence and low stability, the system refers to the secondary model for assistance. To address the mismatch between the four emotion classes and three sentiment categories, we introduce a sentiment mapping strategy: Positive and Neutral map directly to Happy and Neutral, while Negative is classified as Angry or Sad based on the primary model’s confidence.

We evaluate these strategies on the IEMOCAP [1] and MSP-IMPROV [19] datasets to assess their impact on emotion classification performance. By incorporating the secondary model as a fallback mechanism for low-confidence predictions, we observed consistent improvements across both datasets, demonstrating enhanced classification precision and robustness, particularly in emotionally ambiguous cases. These results suggest that our late fusion framework offers a practical and reliable enhancement over a single-modality framework.

2 Proposed Method

This section details the proposed multimodal speech recognition approach, illustrated in Figure 1, which comprises two independent pipelines processing the same audio input222Note that we could directly provide the authentic transcript. However, since obtaining the real transcript is labor-intensive in real-world applications, we assume that only speech is provided in this work, and the transcription is generated by a speech-to-text model. to obtain predictions from two models, followed by an entropy-aware score selection strategy to boost performance.

2.1 Primary Speech Modality

The primary pipeline follows a conventional speech emotion recognition approach. A speech utterance is fed into a self-supervised learning-based wav2vec2 [5] as a feature extractor to obtain emotion-discriminative features, which are then passed through a classifier consisting of two linear projection layers to generate emotion prediction. Specifically, we fine-tune wav2vec2 on the emotion datasets, with its parameters updated during the fine-tuning stage. The primary score can be represented as: $\mathbf{p}^{s}=\{p^{s}_{c}\mid c\in\{\text{Ang, Sad, Hap, Neu}\}\}$, where $p^{s}_{c}$ represents the predicted probability for class $c$ among the four possible emotion classes in the primary branch. Since this branch is trained directly on speech data with labels, we consider it the primary/more reliable source for emotion prediction.

2.2 Secondary Text Modality

The secondary pipeline consists of a Speech-To-Text (S2T) model that generates transcribed text, which is then cleaned and preprocessed through several steps, including expanding contractions, removing punctuation, lemmatizing, lowercasing, and removing numbers, before being passed to a text-based sentiment analysis model [20]. This process does not involve any training on the specific speech datasets, using pretrained S2T and sentiment models applied off-the-shelf provides a second opinion on the expressed sentiment of the input speech.

2.2.1 Whisper Series Speech-to-text Model

Whisper is a robust multilingual transformer-based ASR model developed by OpenAI [21]. It is composed of an encoder-decoder architecture based on the Transformer framework. The encoder ingests log-Mel spectrogram features and consists of multiple stacked Transformer layers with multi-head self-attention, positional encodings, and layer normalization. The decoder generates the text output autoregressively, conditioned on both audio features and previously generated tokens.

2.2.2 RoBERTa Series Sentiment Model

We employ the RoBERTa series model for sentiment analysis [22]. The model architecture consists of Transformer blocks, each with multi-head self-attention, feed-forward layers, residual connections, and layer normalization. The final hidden state is passed through a linear projection layer followed by softmax function to output probabilities across the three sentiment classes: Negative, Neutral, and Positive, denoted as $\mathbf{p}^{t}=\{p^{t}_{i}\mid i\in\{\text{Negative, Neutral, Positive}\}\}$, where $p^{t}_{i}$ represents the predicted probability for class $i$ among the three possible sentiment classes in the secondary branch. This model is directly adopted from an open-source checkpoint trained on large-scale sentiment data. It is used without any adaptation or finetuning to provide a secondary opinion in score fusion.

2.3 Entropy-Aware Score Selection Strategy

After obtaining speech emotion $\mathbf{p}^{s}$ and sentiment $\mathbf{p}^{t}$ scores, an entropy-aware score selection strategy is introduced to determine when the primary model’s predictions should be supplemented by those from the secondary model. The principle is that if the primary model’s prediction has higher certainty, it is retained, otherwise, the system relies on the secondary model.

We utilize two metrics to quantify the certainty of the primary model’s prediction. The first metric is entropy $\mathcal{H}$ , directly measures the degree of uncertainty:

[TABLE]

A lower $\mathcal{H}$ is preferred, indicating high prediction certainty.

The second metric is varentropy $\mathcal{V}$, which quantifies the dispersion of the class probabilities relative to the entropy:

[TABLE]

Varentropy provides an additional measure of confidence by assessing how sharply peaked or flat the probability distribution is, thereby quantifying the stability of entropy $\mathcal{H}$. A higher varentropy $\mathcal{V}$ is preferred, indicating stable uncertainty estimates of the primary model, demonstrating robustness to minor input variations.

If a primary score $\mathbf{p}^{s}$ has high entropy $\mathcal{H}$ and low varentropy $\mathcal{V}$, measured against two thresholds $\tau_{\mathcal{H}}$ and $\tau_{\mathcal{V}}$, it is deemed unreliable. In such cases, we defer the final decision to the secondary sentiment model. This decision is implemented through the merge function described in Algorithm 1, which maps the sentiment prediction into one of the target emotion classes. If the thresholds are not met, the original prediction from the primary model is used.

To derive the optimal thresholds $\tau_{e}$ and $\tau_{v}$, we perform a grid search over all possible pairs of entropy and varentropy values within empirically determined ranges on the training data. Note that the search for optimal thresholds is conducted separately for each emotion class $c$, resulting in four sets of thresholds $\{\tau_{e}^{c},\tau_{v}^{c}\}\in[\{\tau_{e}^{\text{Ang}},\tau_{v}^{\text{Ang}}\},\{\tau_{e}^{\text{Sad}},\tau_{v}^{\text{Sad}}\},\{\tau_{e}^{\text{Hap}},\tau_{v}^{\text{Hap}}\},\{\tau_{e}^{\text{Neu}},\tau_{v}^{\text{Neu}}\}]$.333We initially experimented with a single set of thresholds searched across the entire dataset, but the results were unsatisfactory. Upon analysis, we found that each class exhibits distinct entropy and varentropy distributions, which motivated the use of class-wise thresholds. To do this, we group the training samples based on their emotion labels and determine the thresholds for each group independently. Taking one emotion class $c$ as an example, suppose we have $N$ training samples belonging to class $c$. Let $P_{k}(\mathcal{H})$ and $P_{k}(\mathcal{V})$ denote the $k$-th percentiles of entropy and varentropy values for the $N$ samples, respectively. The search ranges for the thresholds are defined as:

[TABLE]

These ranges are motivated by the observation that incorrect predictions tend to exhibit higher entropy and lower varentropy, with optimal threshold values typically located near the 75th percentile for entropy and the 25th percentile for varentropy. The search is conducted using a fixed step size of $\Delta=\pm 10$ percentile points. Each candidate threshold pair is given by:

[TABLE]

for $k=0,1,\dots,K$ and $l=0,1,\dots,L$, forming a grid of candidate thresholds. The optimal thresholds $(\tau_{\text{e}}^{c},\tau_{\text{v}}^{c})$ for each emotion class $c$ are determined by maximizing a detection accuracy metric $\mathcal{M}$, defined as:

[TABLE]

where the objective metric $\mathcal{M}$ is computed as:

[TABLE]

where $D$ denoting the number of misclassified samples successfully identified by the current thresholding rule, and $T$ representing the total number of samples that satisfy both the entropy and varentropy threshold conditions.

2.4 Sentiment Mapping Strategy

This component addresses the challenge of mapping three-class sentiment outputs (Positive, Neutral, Negative) into the required four emotional classes (Happy, Neutral, Sad, Angry). While positive and neutral sentiments map straightforwardly to Happy and Neutral, respectively, negative sentiment needs further discrimination between Sad and Angry emotions. We propose two methods for this mapping, both of which are evaluated and the one yielding the higher accuracy is selected automatically.

2.4.1 Refer to Primary Model Mapping

In cases of negative sentiment, this method consults the primary model’s confidence scores between Angry and Sad emotions, assigning the sentiment accordingly based on the higher confidence score from the primary model.

2.4.2 Simple or Flip Mapping

We establish a threshold (ranging between 0 and 1) $\tau_{m}^{c}$ for each emotion class $c$ where sentiment scores below the threshold map to Sad and those above to Angry. This mapping is flexible and can be reversed through a ”flip” flag since both emotions can validly represent a negative sentiment. In a similar process to obtaining the optimal entropy and varentropy threshold values, a grid search on the training dataset is performed on a range of discreet values set at constant intervals using the same detection accuracy metric previously mentioned.

2.5 Revert Change Strategy

Once all optimal threshold pairs $\{\tau_{e}^{c},\tau_{v}^{c},\tau_{m}^{c}\}$ are determined, we adopt an additional strategy to enhance the prediction: we aim to construct an exclusion list $\mathcal{E}$ using the training set, where all detrimental emotion changes are stored. Since these changes cause performance drops on the training set, during the inference stage, if such a change occurs, it will be skipped, and the primary prediction will be retained.

Regarding the creation of $\mathcal{E}$, for each training sample, if, for example, a sample’s primary prediction is Angry, and after applying the entropy and valentropy thresholds, the prediction changes to Sad (indicated as AngSad), while the accuracy calculated using the same performance metric defined in Equation 7 drops, this is considered a harmful change and will be added to $\mathcal{E}$. Such changes should be avoided during the reference stage.

3 Experiments

In this section, we first select the speech-to-text model and sentiment model from various publicly available options, and then verify the effectiveness of the proposed entropy-aware score selection method on two commonly-used speech emotion recognition datasets.

3.1 Datasets

Two widely recognized datasets are used to verify the effectiveness of the proposed method. The first is IEMOCAP [1], which comprises 5 sessions, each containing one male and one female actor performing scripted and improvised scenarios. We employ 10-fold cross-validation, where 4 sessions are used for training, and the utterances from the remaining session—containing two speakers—are used for validation and testing, respectively.

The second dataset, MSP-IMPROV [19], consists of 6 sessions, each with one male and one female actor. We adopt 6-fold cross-validation, where each fold designates one complete session as the test set, while the remaining five sessions are split into training (80%) and validation (20%) subsets.

Table 1 lists the number of utterances for each emotion in each session for both datasets.

3.1.1 Settings

For each cross-validation fold, we dynamically determine the optimal entropy, varentropy, and sentiment thresholds based on the training data of that fold, and then apply them to the corresponding test set. This approach ensures adaptive yet consistent prediction merging across all folds for both the IEMOCAP and MSP-IMPROV datasets.

3.2 Results on Various Text Modality Models

The left side of Table 2 compares three S2T ASR models: w2v-CTC [5], Whisper-tiny-en [23], and Whisper-large-v3 [23]. Each model was used to transcribe audio segments from the IEMOCAP dataset. Among them, Whisper-large-v3 achieved the lowest word error rate (WER) of 14.71% and was selected as the primary S2T model for the secondary branch.

The right side of Table 2 compares three sentiment analysis models used to classify authentic IEMOCAP transcripts into three sentiment categories: Positive, Neutral, and Negative, where the Negative class encompasses both angry and sad emotional categories. To evaluate sentiment prediction performance, we tested three advanced Transformer-based models: DistilBERT [24], RoBERTa [25], and RoBERTa-XLM [26]. Cleaned transcripts were passed through each model to generate sentiment predictions. While RoBERTa achieved the highest overall F1 score (46.12%), RoBERTa-XLM showed the most consistent and balanced classification performance across all sentiment categories, particularly excelling in the Negative class with an F1 score of 50.74%. Given the importance of accurately detecting negative sentiment in emotion recognition, RoBERTa-XLM was selected as the sentiment analysis model for our secondary pipeline101010We also experimented with other sentiment models, such as Gemini-1.5-Flash and GPT-4o Mini, but did not observe improved performance. Since these models have not been adapted to IEMOCAP, MSP-IMPROV, or similar content-specific datasets, their performance remains suboptimal. We believe that a sentiment model better tuned to such datasets would yield improved results in the merged pipeline..

3.3 Results on the Entropy-Aware Score Selection

In this section, we first present the averaged results across different folds on the IEMOCAP dataset to compare the effectiveness of using entropy and varentropy thresholds for guiding dynamic switching between the primary and secondary pipeline models, versus not using any score selection. We then report the results for three metrics, Unweighted Accuracy (UA), Weighted Accuracy (WA), and F1 Score, for each fold on both IEMOCAP and MSP-IMPROV.

3.3.1 Comparison With and Without Score Merging Methods

Table 3 lists the results on IEMOCAP with and without score selection strategies. Apparently, applying any score selection strategy, whether based on entropy, varentropy, or both of them, leads to improvements across all evaluation metrics compared to the speech-modality-only method (w/o score selection), except when using entropy alone, where the WA result (63.85%) falls slightly short. Among the strategies, the combined Entropy + Varentropy approach achieves the highest overall performance, with UA (65.81%), WA (65.41%), and F1 score (64.55%), surpassing both individual-threshold methods and significantly outperforming the baseline. Therefore, we adopt the combined approach for its enhanced robustness and reliability for the following experiments.

3.3.2 IEMOCAP Dataset Results

Table 4 shows fold-wise performance of the proposed score selection method on IEMOCAP. Results are compared before and after applying the entropy + varentropy-based score selection. Seven out of ten folds (Folds 3–7, 9, and 10) exhibit consistent improvements across all three evaluation metrics, demonstrating that the merge strategy, through confidence-driven switching, leveraged textual sentiment cues to correct misclassifications. Even in folds like Fold 6 with strong baseline accuracy, the algorithm yielded marginal improvements or stable results, demonstrating non-destructive behavior.

3.3.3 MSP-IMPROV Dataset Results

To further validate the generalizability of our approach, we applied the proposed score selection method to the MSP-IMPROV dataset. The speech scores were obtained from the wav2vec2 model fine-tuned on the MSP-IMPROV dataset, while the text scores were obtained using the same text model as used for IEMOCAP. Results from Table 5 similarly show improvements. Folds 2-4, and 6 in particular saw significant uplifts, suggesting that the entropy + varentropy-based merge is not overfit to IEMOCAP’s structure and retains robustness under distributional shift. The merge mechanism was able to dynamically defer to the secondary sentiment pipeline when confidence was low, preserving precision without introducing instability.

The final assessment shows consistent performance gains using our entropy-aware score selection strategy across both datasets. Although some folds experienced slight performance drops, this may be due to low-confidence predictions or threshold misalignment in certain emotion classes. On average, improvements in F1 score range from 0.5% to 1.2%, indicating the effectiveness of the merging framework while highlighting room for further enhancement. Future work could explore more powerful text-based models to strengthen the utility of the secondary sentiment signal. Additionally, we observe that entropy and varentropy thresholds differ across emotion classes, suggesting that class-specific or dynamically adaptive thresholding could further refine fusion decisions.

3.3.4 Further Analysis

As explained in Section 2.3, lower entropy and higher varentropy are indicative of more accurate predictions, and the search space of these metrics is dependent on the emotion class. Therefore, it is insightful to visualize how their distributions vary across classes. Figure 2 plots the entropy (solid color-filled bars) and varentropy (hatched color-filled bars) for each predicted class on Fold 3 of the IEMOCAP dataset. For each class, the values are shown separately for correctly predicted (green) and incorrectly predicted (red) samples. It is obvious that lower entropy and higher varentropy correlate with more accurate predictions. For entropy, the green bars (correct predictions) are consistently lower than the red bars (incorrect predictions) across all four classes, suggesting that lower entropy values are associated with greater confidence and correctness. In the case of varentropy, the green hatched bars (correct predictions) are generally higher or more centrally distributed than the red hatched bars (incorrect predictions), indicating that higher varentropy values likewise correspond to greater prediction confidence and accuracy.

Moreover, each emotion class demonstrates a distinct range of entropy and varentropy values, reinforcing the need for a per-class thresholding strategy. A single global threshold would fail to capture these class-specific patterns effectively. Additionally, the merging process can occasionally turn a correct prediction into an incorrect one. Therefore, choosing an appropriate threshold becomes a trade-off: it must balance maximizing the detection of incorrect predictions while minimizing the erroneous rejection of correct ones. This trade-off motivates the use of accuracy as the evaluation metric during parameter selection.

Conclusion

In this study, we proposed a multimodal score selection methodology that combines a primary wav2vec2-based speech branch with a secondary Whisper + RoBERTa-XLM sentiment branch for SER. For score selection between the two branches, we leveraged entropy and varentropy thresholds to identify uncertain predictions and dynamically switched to the secondary pipeline for improved reliability. Experiments on both the IEMOCAP and MSP-IMPROV datasets demonstrated clear improvements in accuracy, robustness, and stability across emotional classes, particularly in challenging cases involving conflicting emotional cues. Overall, the proposed late fusion strategy offers a computationally efficient, flexible, and reliable pipeline, outperforming single-modality systems and fixed fusion strategies.