Shapley Value-based Contrastive Alignment for Multimodal Information Extraction

TL;DR

This paper introduces a novel multimodal information extraction approach that uses Shapley value-based contrastive alignment to better bridge semantic and modality gaps between images and text, significantly improving performance.

Contribution

It proposes a new paradigm of Image-Context-Text interaction and a Shapley Value-based Contrastive Alignment method for enhanced multimodal alignment.

Findings

Outperforms state-of-the-art methods on four MIE datasets.

Effectively bridges semantic and modality gaps in multimodal data.

Enhances alignment accuracy through Shapley value assessment and contrastive learning.

Abstract

The rise of social media and the exponential growth of multimodal communication necessitates advanced techniques for Multimodal Information Extraction (MIE). However, existing methodologies primarily rely on direct Image-Text interactions, a paradigm that often faces significant challenges due to semantic and modality gaps between images and text. In this paper, we introduce a new paradigm of Image-Context-Text interaction, where large multimodal models (LMMs) are utilized to generate descriptive textual context to bridge these gaps. In line with this paradigm, we propose a novel Shapley Value-based Contrastive Alignment (Shap-CA) method, which aligns both context-text and context-image pairs. Shap-CA initially applies the Shapley value concept from cooperative game theory to assess the individual contribution of each element in the set of contexts, texts and images towards total semantic and modality overlaps. Following this quantitative evaluation, a contrastive learning strategy is employed to enhance the interactive contribution within context-text/image pairs, while minimizing the influence across these pairs. Furthermore, we design an adaptive fusion module for selective cross-modal fusion. Extensive experiments across four MIE datasets demonstrate that our method significantly outperforms existing state-of-the-art methods.