Explore the Potential of LLMs in Misinformation Detection: An Empirical Study

TL;DR

This paper empirically investigates the effectiveness of Large Language Models in detecting misinformation, revealing their strengths in content analysis and limitations in understanding propagation, with potential to improve existing detection methods.

Contribution

It is the first comprehensive empirical study assessing LLMs' performance on both content and propagation aspects of misinformation detection.

Findings

LLMs perform well in text-based misinformation detection.

LLMs have limited capabilities in understanding propagation structures.

LLMs can enhance existing misinformation detection models.

Abstract

Large Language Models (LLMs) have garnered significant attention for their powerful ability in natural language understanding and reasoning. In this paper, we present a comprehensive empirical study to explore the performance of LLMs on misinformation detection tasks. This study stands as the pioneering investigation into the understanding capabilities of multiple LLMs regarding both content and propagation across social media platforms. Our empirical studies on eight misinformation detection datasets show that LLM-based detectors can achieve comparable performance in text-based misinformation detection but exhibit notably constrained capabilities in comprehending propagation structure compared to existing models in propagation-based misinformation detection. Our experiments further demonstrate that LLMs exhibit great potential to enhance existing misinformation detection models. These findings highlight the potential ability of LLMs to detect misinformation.