MissMAC-Bench: Building Solid Benchmark for Missing Modality Issue in Robust Multimodal Affective Computing

TL;DR

MissMAC-Bench is a comprehensive benchmark designed to evaluate and improve the robustness of multimodal affective computing models in scenarios with missing modality data, addressing a key real-world challenge.

Contribution

This work introduces MissMAC-Bench, a novel benchmark with evaluation protocols for handling incomplete modalities, promoting fair comparison and robustness in MAC models.

Findings

Diverse MAC approaches are validated on 4 datasets with 3 language models.

The benchmark effectively measures model performance under missing modality conditions.

Results highlight the importance of models capable of handling both complete and incomplete data.

Abstract

As a knowledge discovery task over heterogeneous data sources, current Multimodal Affective Computing (MAC) heavily rely on the completeness of multiple modalities to accurately understand human's affective state. However, in real-world scenarios, the availability of modality data is often dynamic and uncertain, leading to substantial performance fluctuations due to the distribution shifts and semantic deficiencies of the incomplete multimodal inputs. Known as the missing modality issue, this challenge poses a critical barrier to the robustness and practical deployment of MAC models. To systematically quantify this issue, we introduce MissMAC-Bench, a comprehensive benchmark designed to establish fair and unified evaluation standards from the perspective of cross-modal synergy. Two guiding principles are proposed, including no missing prior during training, and one single model capable of handling both complete and incomplete modality scenarios, thereby ensuring better generalization. Moreover, to bridge the gap between academic research and real-world applications, our benchmark integrates evaluation protocols with both fixed and random missing patterns at the dataset and instance levels. Extensive experiments conducted on 3 widely-used language models across 4 datasets validate the effectiveness of diverse MAC approaches in tackling the missing modality issue. Our benchmark provides a solid foundation for advancing robust multimodal affective computing and promotes the development of multimedia data mining.