All rivers run into the sea: Unified Modality Brain-like Emotional Central Mechanism
Xinji Mai, Junxiong Lin, Haoran Wang, Zeng Tao, Yan Wang, Shaoqi Yan,, Xuan Tong, Jiawen Yu, Boyang Wang, Ziheng Zhou, Qing Zhao, Shuyong Gao and, Wenqiang Zhang

TL;DR
This paper introduces UMBEnet, a brain-inspired multimodal affective processing network that effectively integrates sensory data for emotion recognition, outperforming existing methods especially under modality missingness.
Contribution
The paper presents UMBEnet, a novel unified modal affective network with a Dual-Stream structure, Prompt Pool, and Sparse Feature Fusion, inspired by brain mechanisms and cross-modal plasticity.
Findings
UMBEnet outperforms state-of-the-art methods on DFER datasets.
It demonstrates high adaptability in modality missing scenarios.
The model effectively fuses multimodal emotional information.
Abstract
In the field of affective computing, fully leveraging information from a variety of sensory modalities is essential for the comprehensive understanding and processing of human emotions. Inspired by the process through which the human brain handles emotions and the theory of cross-modal plasticity, we propose UMBEnet, a brain-like unified modal affective processing network. The primary design of UMBEnet includes a Dual-Stream (DS) structure that fuses inherent prompts with a Prompt Pool and a Sparse Feature Fusion (SFF) module. The design of the Prompt Pool is aimed at integrating information from different modalities, while inherent prompts are intended to enhance the system's predictive guidance capabilities and effectively manage knowledge related to emotion classification. Moreover, considering the sparsity of effective information across different modalities, the SSF module aims to…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsCognitive Abilities and Testing
