Enhancing Multimodal Misinformation Detection by Replaying the Whole Story from Image Modality Perspective

TL;DR

This paper introduces RETSIMD, a novel multimodal misinformation detection method that enhances text analysis by generating auxiliary images from text segments and employs graph neural networks for feature fusion, improving detection accuracy.

Contribution

The paper proposes a new approach that leverages text segmentation, text-to-image generation, and graph neural networks to improve multimodal misinformation detection.

Findings

RETSIMD outperforms existing methods in accuracy.

Generated images from text segments aid in better detection.

Graph-based feature fusion enhances model performance.

Abstract

Multimodal Misinformation Detection (MMD) refers to the task of detecting social media posts involving misinformation, where the post often contains text and image modalities. However, by observing the MMD posts, we hold that the text modality may be much more informative than the image modality because the text generally describes the whole event/story of the current post but the image often presents partial scenes only. Our preliminary empirical results indicate that the image modality exactly contributes less to MMD. Upon this idea, we propose a new MMD method named RETSIMD. Specifically, we suppose that each text can be divided into several segments, and each text segment describes a partial scene that can be presented by an image. Accordingly, we split the text into a sequence of segments, and feed these segments into a pre-trained text-to-image generator to augment a sequence of images. We further incorporate two auxiliary objectives concerning text-image and image-label mutual information, and further post-train the generator over an auxiliary text-to-image generation benchmark dataset. Additionally, we propose a graph structure by defining three heuristic relationships between images, and use a graph neural network to generate the fused features. Extensive empirical results validate the effectiveness of RETSIMD.