Mixed Signals: Understanding Model Disagreement in Multimodal Empathy Detection

TL;DR

This paper investigates why multimodal empathy detection models sometimes fail when different modalities conflict, revealing that disagreements often stem from ambiguity and that disagreement can help identify challenging cases.

Contribution

It provides a detailed analysis of model disagreement in multimodal empathy detection, highlighting the role of ambiguity and proposing disagreement as a diagnostic tool.

Findings

Disagreements often reflect underlying ambiguity and annotator uncertainty.

Dominant modality signals can mislead fusion models when unsupported by others.

Humans and models both show inconsistent benefits from multimodal input.

Abstract

Multimodal models play a key role in empathy detection, but their performance can suffer when modalities provide conflicting cues. To understand these failures, we examine cases where unimodal and multimodal predictions diverge. Using fine-tuned models for text, audio, and video, along with a gated fusion model, we find that such disagreements often reflect underlying ambiguity, as evidenced by annotator uncertainty. Our analysis shows that dominant signals in one modality can mislead fusion when unsupported by others. We also observe that humans, like models, do not consistently benefit from multimodal input. These insights position disagreement as a useful diagnostic signal for identifying challenging examples and improving empathy system robustness.