The Impact of Data Characteristics on GNN Evaluation for Detecting Fake News

TL;DR

This paper critically evaluates popular fake news detection datasets and finds they are inadequate for testing the structural modeling capabilities of GNNs, as the datasets lack meaningful graph diversity and structure plays a negligible role.

Contribution

The study systematically benchmarks GNNs against MLPs on fake news datasets and demonstrates the limited impact of graph structure in these benchmarks, highlighting the need for richer datasets.

Findings

MLPs perform similarly to GNNs on current benchmarks.

Performance drops when node features are shuffled, but not with edge randomization.

Most nodes are only one hop from the root, indicating minimal structural diversity.

Abstract

Graph neural networks (GNNs) are widely used for the detection of fake news by modeling the content and propagation structure of news articles on social media. We show that two of the most commonly used benchmark data sets - GossipCop and PolitiFact - are poorly suited to evaluating the utility of models that use propagation structure. Specifically, these data sets exhibit shallow, ego-like graph topologies that provide little or no ability to differentiate among modeling methods. We systematically benchmark five GNN architectures against a structure-agnostic multilayer perceptron (MLP) that uses the same node features. We show that MLPs match or closely trail the performance of GNNs, with performance gaps often within 1-2% and overlapping confidence intervals. To isolate the contribution of structure in these datasets, we conduct controlled experiments where node features are shuffled or edge structures randomized. We find that performance collapses under feature shuffling but remains stable under edge randomization. This suggests that structure plays a negligible role in these benchmarks. Structural analysis further reveals that over 75% of nodes are only one hop from the root, exhibiting minimal structural diversity. In contrast, on synthetic datasets where node features are noisy and structure is informative, GNNs significantly outperform MLPs. These findings provide strong evidence that widely used benchmarks do not meaningfully test the utility of modeling structural features, and they motivate the development of datasets with richer, more diverse graph topologies.