Information Screening whilst Exploiting! Multimodal Relation Extraction with Feature Denoising and Multimodal Topic Modeling

TL;DR

This paper introduces a novel multimodal relation extraction framework that combines internal-information screening through graph-based denoising and external-information exploitation via multimodal topic modeling, significantly improving performance on benchmark datasets.

Contribution

The proposed framework uniquely integrates graph-based feature denoising with multimodal topic modeling for enhanced relation extraction.

Findings

Outperforms current best models on benchmark datasets

Effectively denoises less-informative features using graph information bottleneck

Enriches context with latent multimodal topic features

Abstract

Existing research on multimodal relation extraction (MRE) faces two co-existing challenges, internal-information over-utilization and external-information under-exploitation. To combat that, we propose a novel framework that simultaneously implements the idea of internal-information screening and external-information exploiting. First, we represent the fine-grained semantic structures of the input image and text with the visual and textual scene graphs, which are further fused into a unified cross-modal graph (CMG). Based on CMG, we perform structure refinement with the guidance of the graph information bottleneck principle, actively denoising the less-informative features. Next, we perform topic modeling over the input image and text, incorporating latent multimodal topic features to enrich the contexts. On the benchmark MRE dataset, our system outperforms the current best model significantly. With further in-depth analyses, we reveal the great potential of our method for the MRE task. Our codes are open at https://github.com/ChocoWu/MRE-ISE.