Mapping (Dis-)Information Flow about the MH17 Plane Crash

TL;DR

This study evaluates the use of neural text classifiers to label and analyze pro-Russian and pro-Ukrainian Twitter content about the MH17 crash, highlighting challenges and potential improvements in disinformation research.

Contribution

It demonstrates the application of neural classifiers for large-scale content labeling in disinformation studies and provides insights into their limitations and error analysis.

Findings

Neural classifiers outperform hashtag-based baselines.

High-precision labeling of pro-Russian and pro-Ukrainian content remains challenging.

Classifier aids human annotation tasks.

Abstract

Digital media enables not only fast sharing of information, but also disinformation. One prominent case of an event leading to circulation of disinformation on social media is the MH17 plane crash. Studies analysing the spread of information about this event on Twitter have focused on small, manually annotated datasets, or used proxys for data annotation. In this work, we examine to what extent text classifiers can be used to label data for subsequent content analysis, in particular we focus on predicting pro-Russian and pro-Ukrainian Twitter content related to the MH17 plane crash. Even though we find that a neural classifier improves over a hashtag based baseline, labeling pro-Russian and pro-Ukrainian content with high precision remains a challenging problem. We provide an error analysis underlining the difficulty of the task and identify factors that might help improve classification in future work. Finally, we show how the classifier can facilitate the annotation task for human annotators.