Follow the Clues, Frame the Truth: Hybrid-evidential Deductive Reasoning in Open-Vocabulary Multimodal Emotion Recognition

TL;DR

HyDRA introduces a hybrid-evidential deductive reasoning framework for open-vocabulary multimodal emotion recognition, effectively reconciling ambiguous cues and outperforming baselines with interpretable evidence tracing.

Contribution

It proposes a novel Hybrid-evidential Deductive Reasoning Architecture that integrates abductive inference with reinforcement learning for improved emotion recognition.

Findings

HyDRA outperforms strong baselines in ambiguous scenarios.

It provides interpretable evidence traces for reasoning.

Reinforcement learning aligns reasoning with task performance.

Abstract

Open-Vocabulary Multimodal Emotion Recognition (OV-MER) is inherently challenging due to the ambiguity of equivocal multimodal cues, which often stem from distinct unobserved situational dynamics. While Multimodal Large Language Models (MLLMs) offer extensive semantic coverage, their performance is often bottlenecked by premature commitment to dominant data priors, resulting in suboptimal heuristics that overlook crucial, complementary affective cues across modalities. We argue that effective affective reasoning requires more than surface-level association; it necessitates reconstructing nuanced emotional states by synthesizing multiple evidence-grounded rationales that reconcile these observations from diverse latent perspectives. We introduce HyDRA, a Hybrid-evidential Deductive Reasoning Architecture that formalizes inference as a Propose-Verify-Decide protocol. To internalize this abductive process, we employ reinforcement learning with hierarchical reward shaping, aligning the reasoning trajectories with final task performance to ensure they best reconcile the observed multimodal cues. Systematic evaluations validate our design choices, with HyDRA consistently outperforming strong baselines--especially in ambiguous or conflicting scenarios--while providing interpretable, diagnostic evidence traces.