A Revisit of Fake News Dataset with Augmented Fact-checking by ChatGPT

TL;DR

This paper revisits fake news datasets by augmenting them with fact-checking from ChatGPT, analyzing differences with human judgments to improve detection and reduce bias.

Contribution

It introduces ChatGPT-FC, an augmented fake news dataset, and provides a quantitative analysis of differences between human and LLM assessments.

Findings

LLM can serve as a preliminary screening tool.

ChatGPT-FC reduces bias in fake news datasets.

LLM assessments differ in credibility and framing evaluations.

Abstract

The proliferation of fake news has emerged as a critical issue in recent years, requiring significant efforts to detect it. However, the existing fake news detection datasets are sourced from human journalists, which are likely to have inherent bias limitations due to the highly subjective nature of this task. In this paper, we revisit the existing fake news dataset verified by human journalists with augmented fact-checking by large language models (ChatGPT), and we name the augmented fake news dataset ChatGPT-FC. We quantitatively analyze the distinctions and resemblances between human journalists and LLM in assessing news subject credibility, news creator credibility, time-sensitive, and political framing. Our findings highlight LLM's potential to serve as a preliminary screening method, offering a promising avenue to mitigate the inherent biases of human journalists and enhance fake news detection.