Multi-source Knowledge Enhanced Graph Attention Networks for Multimodal Fact Verification

TL;DR

This paper introduces MultiKE-GAT, a novel multimodal fact verification model that leverages external knowledge and heterogeneous graph attention to improve the fusion of multiple modalities for assessing claim veracity.

Contribution

The paper proposes a new model that incorporates external multimodal knowledge and constructs a heterogeneous graph to enhance feature fusion in multimodal fact verification.

Findings

Outperforms existing methods on benchmark datasets

Effectively fuses multimodal features with external knowledge

Reduces noise and inconsistencies in evidence analysis

Abstract

Multimodal fact verification is an under-explored and emerging field that has gained increasing attention in recent years. The goal is to assess the veracity of claims that involve multiple modalities by analyzing the retrieved evidence. The main challenge in this area is to effectively fuse features from different modalities to learn meaningful multimodal representations. To this end, we propose a novel model named Multi-Source Knowledge-enhanced Graph Attention Network (MultiKE-GAT). MultiKE-GAT introduces external multimodal knowledge from different sources and constructs a heterogeneous graph to capture complex cross-modal and cross-source interactions. We exploit a Knowledge-aware Graph Fusion (KGF) module to learn knowledge-enhanced representations for each claim and evidence and eliminate inconsistencies and noises introduced by redundant entities. Experiments on two public benchmark datasets demonstrate that our model outperforms other comparison methods, showing the effectiveness and superiority of the proposed model.