How Good Are SOTA Fake News Detectors

TL;DR

This paper evaluates the robustness of traditional and state-of-the-art fake news detectors, revealing that traditional models often generalize better to unseen data than large language models, with the best choice depending on the task.

Contribution

It provides a comparative analysis of traditional and deep learning models' robustness in fake news detection, highlighting their generalization capabilities.

Findings

Traditional models generalize better to out-of-distribution data.

Large language models perform well on in-distribution data but less so on unseen data.

Model choice depends on specific fake news detection tasks.

Abstract

Automatic fake news detection with machine learning can prevent the dissemination of false statements before they gain many views. Several datasets labeling statements as legitimate or false have been created since the 2016 United States presidential election for the prospect of training machine learning models. We evaluate the robustness of both traditional and deep state-of-the-art models to gauge how well they may perform in the real world. We find that traditional models tend to generalize better to data outside the distribution it was trained on compared to more recently-developed large language models, though the best model to use may depend on the specific task at hand.