Revisiting Fake News Detection: Towards Temporality-aware Evaluation by Leveraging Engagement Earliness

TL;DR

This paper introduces a temporality-aware evaluation scheme for social graph-based fake news detection, highlighting the importance of engagement earliness, and proposes DAWN, a method that improves detection accuracy in realistic, time-sensitive scenarios.

Contribution

The paper formalizes a realistic evaluation setting for fake news detection and proposes DAWN, a novel method leveraging engagement earliness to suppress noisy social graph edges.

Findings

Conventional methods' effectiveness drops under temporality-aware evaluation.

Engagement earliness features help reduce noise in social graph edges.

DAWN outperforms existing methods in real-world, temporally-aware scenarios.

Abstract

Social graph-based fake news detection aims to identify news articles containing false information by utilizing social contexts, e.g., user information, tweets and comments. However, conventional methods are evaluated under less realistic scenarios, where the model has access to future knowledge on article-related and context-related data during training. In this work, we newly formalize a more realistic evaluation scheme that mimics real-world scenarios, where the data is temporality-aware and the detection model can only be trained on data collected up to a certain point in time. We show that the discriminative capabilities of conventional methods decrease sharply under this new setting, and further propose DAWN, a method more applicable to such scenarios. Our empirical findings indicate that later engagements (e.g., consuming or reposting news) contribute more to noisy edges that link real news-fake news pairs in the social graph. Motivated by this, we utilize feature representations of engagement earliness to guide an edge weight estimator to suppress the weights of such noisy edges, thereby enhancing the detection performance of DAWN. Through extensive experiments, we demonstrate that DAWN outperforms existing fake news detection methods under real-world environments. The source code is available at https://github.com/LeeJunmo/DAWN.