Conflict-Aware Multimodal Fusion for Ambivalence and Hesitancy Recognition

TL;DR

This paper introduces ConflictAwareAH, a multimodal framework that detects ambivalence and hesitancy by analyzing disagreements across speech, face, and text signals, significantly improving recognition accuracy.

Contribution

It proposes a conflict-aware multimodal fusion method using pairwise differences to better recognize subtle affective states, addressing limitations of text-only approaches.

Findings

Improves F1-NoAH by +4.6 points over text-only models.

Halves the performance gap between classes.

Achieves state-of-the-art results on BAH dataset with 0.694 Macro F1.

Abstract

Ambivalence and hesitancy (A/H) are subtle affective states where a person shows conflicting signals through different channels -- saying one thing while their face or voice tells another story. Recognising these states automatically is valuable in clinical settings, but it is hard for machines because the key evidence lives in the \emph{disagreements} between what is said, how it sounds, and what the face shows. We present \textbf{ConflictAwareAH}, a multimodal framework built for this problem. Three pre-trained encoders extract video, audio, and text representations. Pairwise conflict features -- element-wise absolute differences between modality embeddings -- serve as \emph{bidirectional} cues: large cross-modal differences flag A/H, while small differences confirm behavioural consistency and anchor the negative class. This conflict-aware design addresses a key limitation of text-dominant approaches, which tend to over-detect A/H (high F1-AH) while struggling to confirm its absence: our multimodal model improves F1-NoAH by +4.6 points over text alone and halves the class-performance gap. A complementary \emph{text-guided late fusion} strategy blends a text-only auxiliary head with the full model at inference, adding +4.1 Macro F1. On the BAH dataset from the ABAW10 Ambivalence/Hesitancy Challenge, our method reaches \textbf{0.694 Macro F1} on the labelled test split and \textbf{0.715} on the private leaderboard, outperforming published multimodal baselines by over 10 points -- all on a single GPU in under 25 minutes of training.