FACTIFY3M: A Benchmark for Multimodal Fact Verification with Explainability through 5W Question-Answering

TL;DR

This paper introduces FACTIFY 3M, a large multimodal dataset for fake news detection that includes explainability features through 5W question-answering, addressing the need for scalable multimodal disinformation verification.

Contribution

The paper presents a novel multimodal fact verification dataset with 3 million samples, incorporating explainability via 5W QA, and includes diverse data such as paraphrased claims, images, and heatmaps.

Findings

First large-scale multimodal fact verification dataset.

Inclusion of explainability features like 5W QA and heatmaps.

Addresses gap in multimodal disinformation detection research.

Abstract

Combating disinformation is one of the burning societal crises -- about 67% of the American population believes that disinformation produces a lot of uncertainty, and 10% of them knowingly propagate disinformation. Evidence shows that disinformation can manipulate democratic processes and public opinion, causing disruption in the share market, panic and anxiety in society, and even death during crises. Therefore, disinformation should be identified promptly and, if possible, mitigated. With approximately 3.2 billion images and 720,000 hours of video shared online daily on social media platforms, scalable detection of multimodal disinformation requires efficient fact verification. Despite progress in automatic text-based fact verification (e.g., FEVER, LIAR), the research community lacks substantial effort in multimodal fact verification. To address this gap, we introduce FACTIFY 3M, a dataset of 3 million samples that pushes the boundaries of the domain of fact verification via a multimodal fake news dataset, in addition to offering explainability through the concept of 5W question-answering. Salient features of the dataset include: (i) textual claims, (ii) ChatGPT-generated paraphrased claims, (iii) associated images, (iv) stable diffusion-generated additional images (i.e., visual paraphrases), (v) pixel-level image heatmap to foster image-text explainability of the claim, (vi) 5W QA pairs, and (vii) adversarial fake news stories.