Automated Claim Matching with Large Language Models: Empowering Fact-Checkers in the Fight Against Misinformation

TL;DR

This paper presents FACT-GPT, a framework that uses large language models to automate claim matching in fact-checking, improving efficiency and supporting fact-checkers against misinformation with publicly available datasets and models.

Contribution

Introduces FACT-GPT, a novel LLM-based framework for automating claim matching in fact-checking, with fine-tuning techniques and publicly released resources.

Findings

Fine-tuned LLMs match larger pre-trained models in claim matching

The framework closely aligns with human annotations

Provides datasets and models for further research

Abstract

In today's digital era, the rapid spread of misinformation poses threats to public well-being and societal trust. As online misinformation proliferates, manual verification by fact checkers becomes increasingly challenging. We introduce FACT-GPT (Fact-checking Augmentation with Claim matching Task-oriented Generative Pre-trained Transformer), a framework designed to automate the claim matching phase of fact-checking using Large Language Models (LLMs). This framework identifies new social media content that either supports or contradicts claims previously debunked by fact-checkers. Our approach employs GPT-4 to generate a labeled dataset consisting of simulated social media posts. This data set serves as a training ground for fine-tuning more specialized LLMs. We evaluated FACT-GPT on an extensive dataset of social media content related to public health. The results indicate that our fine-tuned LLMs rival the performance of larger pre-trained LLMs in claim matching tasks, aligning closely with human annotations. This study achieves three key milestones: it provides an automated framework for enhanced fact-checking; demonstrates the potential of LLMs to complement human expertise; offers public resources, including datasets and models, to further research and applications in the fact-checking domain.