AffectVerse: Emotional World Models for Multimodal Affective Computing

TL;DR

AffectVerse is a multimodal affective computing model that predicts future emotional states using a novel belief module, improving emotion recognition across multiple benchmarks.

Contribution

It introduces AffectVerse with an Emotion World Module that models affective dynamics through future prediction, a novel approach in multimodal emotion recognition.

Findings

AffectVerse outperforms existing models by at least 2.57% on nine benchmarks.

Temporal imagination and belief aggregation contribute to performance gains.

Predictive belief-state modeling is effective for affective computing.

Abstract

Humans infer emotions by integrating observed multimodal cues with expectations about how affective states may unfold. Existing multimodal large language models (MLLMs), however, often treat emotion recognition as static fusion over complete audiovisual-text inputs, leaving affective dynamics implicit. We propose AffectVerse, a Qwen2.5-Omni-based model equipped with an Emotion World Module (EWM), an action-free representation-level module for short-horizon latent affective prediction. \rev{EWM contains three modules: 1) Cross-Modal Temporal Imagination predicts future video/audio representations from past tokens with multi-step rollout. 2) MAMA(Modality-Aware Multi-step Attention) Belief Aggregation compresses imagined tokens into modality-aware belief tokens. 3) Belief Injection inserts these belief tokens into the LLM for affective reasoning.} AffectVerse uses future prediction as a past-conditioned self-supervised signal: it does not replace modeling observed history or require unseen signals at inference, but forces the current belief state to encode transition cues that are predictive of subsequent affective change. Across nine benchmarks, AffectVerse improves at least 2.57\% over other models, while controlled ablations show additive gains from temporal imagination, cross-modal rollout, and belief aggregation. These results suggest predictive belief-state modeling is a practical alternative for affective computing.