An exploration of features to improve the generalisability of fake news detection models

TL;DR

This paper investigates how stylistic and social-monetisation features can improve the generalisability of fake news detection models, addressing dataset bias issues and evaluating LLMs like LLaMa.

Contribution

It introduces novel social-monetisation features and demonstrates their effectiveness in enhancing fake news detection robustness over traditional token-based models.

Findings

Stylistic and social-monetisation features outperform token-based models in generalisability.

Token-based models are sensitive to dataset biases and perform poorly across datasets.

LLMs like LLaMa show limited effectiveness compared to feature-based approaches.

Abstract

Fake news poses global risks by influencing elections and spreading misinformation, making detection critical. Existing NLP and supervised Machine Learning methods perform well under cross-validation but struggle to generalise across datasets, even within the same domain. This issue stems from coarsely labelled training data, where articles are labelled based on their publisher, introducing biases that token-based models like TF-IDF and BERT are sensitive to. While Large Language Models (LLMs) offer promise, their application in fake news detection remains limited. This study demonstrates that meaningful features can still be extracted from coarsely labelled data to improve real-world robustness. Stylistic features-lexical, syntactic, and semantic-are explored due to their reduced sensitivity to dataset biases. Additionally, novel social-monetisation features are introduced, capturing economic incentives behind fake news, such as advertisements, external links, and social media elements. The study trains on the coarsely labelled NELA 2020-21 dataset and evaluates using the manually labelled Facebook URLs dataset, a gold standard for generalisability. Results highlight the limitations of token-based models trained on biased data and contribute to the scarce evidence on LLMs like LLaMa in this field. Findings indicate that stylistic and social-monetisation features offer more generalisable predictions than token-based methods and LLMs. Statistical and permutation feature importance analyses further reveal their potential to enhance performance and mitigate dataset biases, providing a path forward for improving fake news detection.