Evaluation of Fake News Detection with Knowledge-Enhanced Language Models

TL;DR

This paper explores how integrating structured factual knowledge into large language models can improve fake news detection, with mixed results depending on the relevance and currency of the knowledge base used.

Contribution

It systematically evaluates the impact of knowledge integration into PLMs for fake news detection across different datasets and knowledge bases.

Findings

Knowledge-enhanced models improve detection on LIAR dataset.

Mixed results on COVID-19 dataset highlight the importance of relevant and current KBs.

Relevance of knowledge bases affects the effectiveness of knowledge integration.

Abstract

Recent advances in fake news detection have exploited the success of large-scale pre-trained language models (PLMs). The predominant state-of-the-art approaches are based on fine-tuning PLMs on labelled fake news datasets. However, large-scale PLMs are generally not trained on structured factual data and hence may not possess priors that are grounded in factually accurate knowledge. The use of existing knowledge bases (KBs) with rich human-curated factual information has thus the potential to make fake news detection more effective and robust. In this paper, we investigate the impact of knowledge integration into PLMs for fake news detection. We study several state-of-the-art approaches for knowledge integration, mostly using Wikidata as KB, on two popular fake news datasets - LIAR, a politics-based dataset, and COVID-19, a dataset of messages posted on social media relating to the COVID-19 pandemic. Our experiments show that knowledge-enhanced models can significantly improve fake news detection on LIAR where the KB is relevant and up-to-date. The mixed results on COVID-19 highlight the reliance on stylistic features and the importance of domain-specific and current KBs.