From Deception to Detection: The Dual Roles of Large Language Models in Fake News

TL;DR

This paper investigates the dual roles of Large Language Models in fake news, analyzing their potential to both generate and detect misinformation, and compares their effectiveness across different models.

Contribution

First comprehensive analysis of seven LLMs' capabilities in fake news generation and detection, highlighting their strengths and limitations.

Findings

Some models refuse to generate biased content due to safety protocols.

Larger models show better fake news detection performance.

LLM-generated fake news are harder to detect than human-written ones.

Abstract

Fake news poses a significant threat to the integrity of information ecosystems and public trust. The advent of Large Language Models (LLMs) holds considerable promise for transforming the battle against fake news. Generally, LLMs represent a double-edged sword in this struggle. One major concern is that LLMs can be readily used to craft and disseminate misleading information on a large scale. This raises the pressing questions: Can LLMs easily generate biased fake news? Do all LLMs have this capability? Conversely, LLMs offer valuable prospects for countering fake news, thanks to their extensive knowledge of the world and robust reasoning capabilities. This leads to other critical inquiries: Can we use LLMs to detect fake news, and do they outperform typical detection models? In this paper, we aim to address these pivotal questions by exploring the performance of various LLMs. Our objective is to explore the capability of various LLMs in effectively combating fake news, marking this as the first investigation to analyze seven such models. Our results reveal that while some models adhere strictly to safety protocols, refusing to generate biased or misleading content, other models can readily produce fake news across a spectrum of biases. Additionally, our results show that larger models generally exhibit superior detection abilities and that LLM-generated fake news are less likely to be detected than human-written ones. Finally, our findings demonstrate that users can benefit from LLM-generated explanations in identifying fake news.