Debiased Multimodal Understanding for Human Language Sequences

TL;DR

This paper introduces a causal intervention module called SuCI to mitigate subject bias in multimodal language understanding, improving model generalization across diverse subjects.

Contribution

It formulates a causal graph for MLU, identifies subject bias as a confounder, and proposes SuCI to disentangle and remove this bias for more unbiased predictions.

Findings

SuCI improves performance on multiple MLU benchmarks.

The method effectively reduces subject-specific biases.

Results demonstrate enhanced generalization to new subjects.

Abstract

Human multimodal language understanding (MLU) is an indispensable component of expression analysis (e.g., sentiment or humor) from heterogeneous modalities, including visual postures, linguistic contents, and acoustic behaviours. Existing works invariably focus on designing sophisticated structures or fusion strategies to achieve impressive improvements. Unfortunately, they all suffer from the subject variation problem due to data distribution discrepancies among subjects. Concretely, MLU models are easily misled by distinct subjects with different expression customs and characteristics in the training data to learn subject-specific spurious correlations, limiting performance and generalizability across new subjects. Motivated by this observation, we introduce a recapitulative causal graph to formulate the MLU procedure and analyze the confounding effect of subjects. Then, we propose SuCI, a simple yet effective causal intervention module to disentangle the impact of subjects acting as unobserved confounders and achieve model training via true causal effects. As a plug-and-play component, SuCI can be widely applied to most methods that seek unbiased predictions. Comprehensive experiments on several MLU benchmarks clearly show the effectiveness of the proposed module.