Avoiding Over-smoothing in Social Media Rumor Detection with Pre-trained Propagation Tree Transformer

TL;DR

This paper introduces P2T3, a Transformer-based approach that effectively detects social media rumors by avoiding over-smoothing issues common in GNNs, leveraging pre-training on large datasets and capturing long-range dependencies.

Contribution

The paper presents P2T3, a novel Transformer architecture tailored for rumor detection that overcomes over-smoothing and long-range dependency challenges in propagation trees.

Findings

P2T3 outperforms previous state-of-the-art methods on multiple benchmarks.

P2T3 performs well in few-shot learning scenarios.

P2T3 effectively captures long-range dependencies in rumor propagation trees.

Abstract

Deep learning techniques for rumor detection typically utilize Graph Neural Networks (GNNs) to analyze post relations. These methods, however, falter due to over-smoothing issues when processing rumor propagation structures, leading to declining performance. Our investigation into this issue reveals that over-smoothing is intrinsically tied to the structural characteristics of rumor propagation trees, in which the majority of nodes are 1-level nodes. Furthermore, GNNs struggle to capture long-range dependencies within these trees. To circumvent these challenges, we propose a Pre-Trained Propagation Tree Transformer (P2T3) method based on pure Transformer architecture. It extracts all conversation chains from a tree structure following the propagation direction of replies, utilizes token-wise embedding to infuse connection information and introduces necessary inductive bias, and pre-trains on large-scale unlabeled datasets. Experiments indicate that P2T3 surpasses previous state-of-the-art methods in multiple benchmark datasets and performs well under few-shot conditions. P2T3 not only avoids the over-smoothing issue inherent in GNNs but also potentially offers a large model or unified multi-modal scheme for future social media research.