PHEMEPlus: Enriching Social Media Rumour Verification with External Evidence

TL;DR

This paper introduces PHEMEPlus, a new dataset combining social media conversations with external web evidence to improve rumour verification, and evaluates methods for effective evidence retrieval.

Contribution

It provides the first dataset integrating social media and external evidence for rumour verification and assesses query formulation techniques for evidence retrieval.

Findings

Incorporating external evidence improves rumour verification accuracy.

Different query formulation methods vary in effectiveness.

PHEMEPlus dataset facilitates future research in social media rumour verification.

Abstract

Work on social media rumour verification utilises signals from posts, their propagation and users involved. Other lines of work target identifying and fact-checking claims based on information from Wikipedia, or trustworthy news articles without considering social media context. However works combining the information from social media with external evidence from the wider web are lacking. To facilitate research in this direction, we release a novel dataset, PHEMEPlus, an extension of the PHEME benchmark, which contains social media conversations as well as relevant external evidence for each rumour. We demonstrate the effectiveness of incorporating such evidence in improving rumour verification models. Additionally, as part of the evidence collection, we evaluate various ways of query formulation to identify the most effective method.