Interpreting and Enhancing Emotional Circuits in Large Vision-Language Models via Cross-Modal Information Flow

TL;DR

This paper investigates how large vision-language models process emotional information, revealing internal mechanisms and proposing interventions to improve emotional reasoning and reduce hallucinations.

Contribution

It introduces a causal attribution framework and a specialized dataset to analyze and enhance emotional circuits in LVLMs, improving their emotional reasoning capabilities.

Findings

Visual emotional cues are aggregated in middle layers via sentiment-specific attention heads.

Deep layers translate emotional cues into narratives through emotion-general pathways.

Inference-time interventions improve model performance and reduce emotional hallucinations.

Abstract

Large Vision-Language Models (LVLMs) represent a significant leap towards empathetic agents, demonstrating remarkable capabilities in emotion understanding. However, the internal mechanisms governing how LVLMs translate abstract visual stimuli into coherent emotional narratives remain largely unexplored, primarily due to the scarcity of visual counterfactuals and the diffuse nature of emotional expression. In this paper, we bridge this gap by introducing a steering-vector-based causal attribution framework tailored for descriptive emotional reasoning. To this end, we construct a specialized dataset to demystify the emotional circuits underlying the three-stage ``Adapt-Aggregate-Execute'' mechanism. Crucially, we discover a functional decoupling: visual emotional cues are aggregated in middle layers via sentiment-specific attention heads, but are subsequently translated into narrative generation in deep layers through emotion-general pathways. Guided by these insights, we regulate the emotional information routing to strengthen attention flow and amplify the semantic activation to consolidate expression. Extensive experiments on the comprehensive MER-UniBench demonstrate that our methods significantly improve performance via inference-time intervention, effectively mitigating emotional hallucinations and corroborating the causal fidelity of the discovered circuits.