MEG: Multi-Evidence GNN for Multimodal Semantic Forensics

TL;DR

This paper introduces MEG, a graph neural network model that leverages multiple evidences for multimodal semantic forensics, significantly improving fake news detection accuracy across various modalities.

Contribution

The paper presents a novel GNN-based approach that effectively utilizes multiple evidences, enhancing scalability and performance in multimodal semantic forensics.

Findings

Outperforms existing methods with up to 25% error reduction.

Effectively utilizes multiple evidences for improved detection.

Scalable with the number of evidences.

Abstract

Fake news often involves semantic manipulations across modalities such as image, text, location etc and requires the development of multimodal semantic forensics for its detection. Recent research has centered the problem around images, calling it image repurposing -- where a digitally unmanipulated image is semantically misrepresented by means of its accompanying multimodal metadata such as captions, location, etc. The image and metadata together comprise a multimedia package. The problem setup requires algorithms to perform multimodal semantic forensics to authenticate a query multimedia package using a reference dataset of potentially related packages as evidences. Existing methods are limited to using a single evidence (retrieved package), which ignores potential performance improvement from the use of multiple evidences. In this work, we introduce a novel graph neural network based model for multimodal semantic forensics, which effectively utilizes multiple retrieved packages as evidences and is scalable with the number of evidences. We compare the scalability and performance of our model against existing methods. Experimental results show that the proposed model outperforms existing state-of-the-art algorithms with an error reduction of up to 25%.