Bridging the Emotional Semantic Gap via Multimodal Relevance Estimation

TL;DR

This paper introduces a multimodal relevance estimation network that uses attention, semantic estimation loss, and contrastive learning to effectively bridge the semantic gap among different emotional modalities, improving emotion recognition accuracy.

Contribution

It proposes a novel multimodal relevance estimation framework with a new dataset, SDME, to better capture relevant semantics across modalities in emotional analysis.

Findings

Effective in capturing relevant semantics despite large deviations

Improves multimodal emotion recognition accuracy

Provides a new dataset for semantic relevance research

Abstract

Human beings have rich ways of emotional expressions, including facial action, voice, and natural languages. Due to the diversity and complexity of different individuals, the emotions expressed by various modalities may be semantically irrelevant. Directly fusing information from different modalities may inevitably make the model subject to the noise from semantically irrelevant modalities. To tackle this problem, we propose a multimodal relevance estimation network to capture the relevant semantics among modalities in multimodal emotions. Specifically, we take advantage of an attention mechanism to reflect the semantic relevance weights of each modality. Moreover, we propose a relevant semantic estimation loss to weakly supervise the semantics of each modality. Furthermore, we make use of contrastive learning to optimize the similarity of category-level modality-relevant semantics across different modalities in feature space, thereby bridging the semantic gap between heterogeneous modalities. In order to better reflect the emotional state in the real interactive scenarios and perform the semantic relevance analysis, we collect a single-label discrete multimodal emotion dataset named SDME, which enables researchers to conduct multimodal semantic relevance research with large category bias. Experiments on continuous and discrete emotion datasets show that our model can effectively capture the relevant semantics, especially for the large deviations in modal semantics. The code and SDME dataset will be publicly available.