Variational Multi-Modal Hypergraph Attention Network for Multi-Modal Relation Extraction

TL;DR

This paper introduces VM-HAN, a novel multi-modal hypergraph attention network that models high-order intra- and inter-modal correlations for improved multi-modal relation extraction, achieving state-of-the-art results.

Contribution

The paper proposes a variational hypergraph attention network that captures diverse entity pair representations and models complex correlations in multi-modal relation extraction.

Findings

Achieves state-of-the-art accuracy on MMRE datasets.

Effectively models high-order intra-/inter-modal correlations.

Improves efficiency over existing methods.

Abstract

Multi-modal relation extraction (MMRE) is a challenging task that aims to identify relations between entities in text leveraging image information. Existing methods are limited by their neglect of the multiple entity pairs in one sentence sharing very similar contextual information (ie, the same text and image), resulting in increased difficulty in the MMRE task. To address this limitation, we propose the Variational Multi-Modal Hypergraph Attention Network (VM-HAN) for multi-modal relation extraction. Specifically, we first construct a multi-modal hypergraph for each sentence with the corresponding image, to establish different high-order intra-/inter-modal correlations for different entity pairs in each sentence. We further design the Variational Hypergraph Attention Networks (V-HAN) to obtain representational diversity among different entity pairs using Gaussian distribution and learn a better hypergraph structure via variational attention. VM-HAN achieves state-of-the-art performance on the multi-modal relation extraction task, outperforming existing methods in terms of accuracy and efficiency.