A Trustworthy Method for Multimodal Emotion Recognition

TL;DR

This paper introduces a trusted emotion recognition framework that uses uncertainty estimation to improve reliability and robustness of multimodal emotion detection, especially under noisy or corrupted data conditions.

Contribution

It proposes a novel trusted emotion recognition (TER) method that incorporates confidence-based fusion and new evaluation metrics for reliability assessment.

Findings

Achieves state-of-the-art accuracy on Music-video dataset.

Outperforms existing methods in trusted F1 scores on IEMOCAP and Music-video.

Demonstrates robustness against noise and data corruption.

Abstract

Existing emotion recognition methods mainly focus on enhancing performance by employing complex deep models, typically resulting in significantly higher model complexity. Although effective, it is also crucial to ensure the reliability of the final decision, especially for noisy, corrupted and out-of-distribution data. To this end, we propose a novel emotion recognition method called trusted emotion recognition (TER), which utilizes uncertainty estimation to calculate the confidence value of predictions. TER combines the results from multiple modalities based on their confidence values to output the trusted predictions. We also provide a new evaluation criterion to assess the reliability of predictions. Specifically, we incorporate trusted precision and trusted recall to determine the trusted threshold and formulate the trusted Acc. and trusted F1 score to evaluate the model's trusted performance. The proposed framework combines the confidence module that accordingly endows the model with reliability and robustness against possible noise or corruption. The extensive experimental results validate the effectiveness of our proposed model. The TER achieves state-of-the-art performance on the Music-video, achieving 82.40% Acc. In terms of trusted performance, TER outperforms other methods on the IEMOCAP and Music-video, achieving trusted F1 scores of 0.7511 and 0.9035, respectively.