Retrieval Augmented Verification for Zero-Shot Detection of Multimodal Disinformation

TL;DR

This paper presents a zero-shot, graph-based framework for detecting multimodal disinformation on social media by analyzing textual and visual evidence, providing interpretability and outperforming some existing methods.

Contribution

The novel approach combines graph representations with pretrained visual features for zero-shot disinformation detection, enhancing transparency and interpretability.

Findings

Achieves competitive performance with state-of-the-art methods.

Provides transparent explanations for disinformation detection.

Effectively analyzes multimodal social media claims in real-time.

Abstract

The rise of disinformation on social media, especially through the strategic manipulation or repurposing of images, paired with provocative text, presents a complex challenge for traditional fact-checking methods. In this paper, we introduce a novel zero-shot approach to identify and interpret such multimodal disinformation, leveraging real-time evidence from credible sources. Our framework goes beyond simple true-or-false classifications by analyzing both the textual and visual components of social media claims in a structured, interpretable manner. By constructing a graph-based representation of entities and relationships within the claim, combined with pretrained visual features, our system automatically retrieves and matches external evidence to identify inconsistencies. Unlike traditional models dependent on labeled datasets, our method empowers users with transparency, illuminating exactly which aspects of the claim hold up to scrutiny and which do not. Our framework achieves competitive performance with state-of-the-art methods while offering enhanced explainability.