Memory-guided Prototypical Co-occurrence Learning for Mixed Emotion Recognition

TL;DR

This paper introduces a novel memory-guided framework for recognizing multiple co-occurring emotions from multi-modal signals, addressing limitations of existing models by modeling emotion co-occurrence patterns and structured correlations.

Contribution

The proposed MPCL framework explicitly models emotion co-occurrence and structured correlations using memory mechanisms, prototype banks, and relation distillation, advancing mixed emotion recognition.

Findings

Outperforms state-of-the-art methods on public datasets

Achieves higher accuracy in mixed emotion recognition

Demonstrates effective modeling of emotion co-occurrence patterns

Abstract

Emotion recognition from multi-modal physiological and behavioral signals plays a pivotal role in affective computing, yet most existing models remain constrained to the prediction of singular emotions in controlled laboratory settings. Real-world human emotional experiences, by contrast, are often characterized by the simultaneous presence of multiple affective states, spurring recent interest in mixed emotion recognition as an emotion distribution learning problem. Current approaches, however, often neglect the valence consistency and structured correlations inherent among coexisting emotions. To address this limitation, we propose a Memory-guided Prototypical Co-occurrence Learning (MPCL) framework that explicitly models emotion co-occurrence patterns. Specifically, we first fuse multi-modal signals via a multi-scale associative memory mechanism. To capture cross-modal semantic relationships, we construct emotion-specific prototype memory banks, yielding rich physiological and behavioral representations, and employ prototype relation distillation to ensure cross-modal alignment in the latent prototype space. Furthermore, inspired by human cognitive memory systems, we introduce a memory retrieval strategy to extract semantic-level co-occurrence associations across emotion categories. Through this bottom-up hierarchical abstraction process, our model learns affectively informative representations for accurate emotion distribution prediction. Comprehensive experiments on two public datasets demonstrate that MPCL consistently outperforms state-of-the-art methods in mixed emotion recognition, both quantitatively and qualitatively.