RAGAR, Your Falsehood Radar: RAG-Augmented Reasoning for Political Fact-Checking using Multimodal Large Language Models

TL;DR

This paper presents RAGAR, a multimodal fact-checking system for political misinformation that uses retrieval-augmented reasoning techniques, achieving high accuracy and human-validated explanations.

Contribution

It introduces two novel reasoning methods, CoRAG and ToRAG, for multimodal fact-checking, enhancing accuracy over baseline approaches.

Findings

Weighted F1-score of 0.85, surpassing baseline by 0.14

Human evaluation confirms explanations include all gold standard info

Effective reasoning on multimodal claims using RAG techniques

Abstract

The escalating challenge of misinformation, particularly in political discourse, requires advanced fact-checking solutions; this is even clearer in the more complex scenario of multimodal claims. We tackle this issue using a multimodal large language model in conjunction with retrieval-augmented generation (RAG), and introduce two novel reasoning techniques: Chain of RAG (CoRAG) and Tree of RAG (ToRAG). They fact-check multimodal claims by extracting both textual and image content, retrieving external information, and reasoning subsequent questions to be answered based on prior evidence. We achieve a weighted F1-score of 0.85, surpassing a baseline reasoning technique by 0.14 points. Human evaluation confirms that the vast majority of our generated fact-check explanations contain all information from gold standard data.