Machine Generation and Detection of Arabic Manipulated and Fake News

TL;DR

This paper introduces a new method for generating and detecting manipulated Arabic news, providing a large dataset and models that improve detection accuracy of fake news in Arabic.

Contribution

It presents a novel Arabic news manipulation generation method, a large POS-tagged dataset, and state-of-the-art detection models for Arabic fake news.

Findings

Human annotation reveals effects of machine manipulation on veracity

Detection models achieve macro F1=70.06, outperforming previous methods

The dataset and models are publicly available for future research

Abstract

Fake news and deceptive machine-generated text are serious problems threatening modern societies, including in the Arab world. This motivates work on detecting false and manipulated stories online. However, a bottleneck for this research is lack of sufficient data to train detection models. We present a novel method for automatically generating Arabic manipulated (and potentially fake) news stories. Our method is simple and only depends on availability of true stories, which are abundant online, and a part of speech tagger (POS). To facilitate future work, we dispense with both of these requirements altogether by providing AraNews, a novel and large POS-tagged news dataset that can be used off-the-shelf. Using stories generated based on AraNews, we carry out a human annotation study that casts light on the effects of machine manipulation on text veracity. The study also measures human ability to detect Arabic machine manipulated text generated by our method. Finally, we develop the first models for detecting manipulated Arabic news and achieve state-of-the-art results on Arabic fake news detection (macro F1=70.06). Our models and data are publicly available.