Ex Machina: Personal Attacks Seen at Scale
Ellery Wulczyn, Nithum Thain, Lucas Dixon

TL;DR
This paper presents a scalable method combining crowdsourcing and machine learning to analyze personal attacks on online platforms, applied to Wikipedia, revealing attack patterns and challenging common assumptions.
Contribution
It introduces a novel evaluation approach for classifiers using crowd-worker aggregation and applies it to large-scale Wikipedia comment analysis.
Findings
Most attacks are not from a few malicious users
Attacks are not mainly due to anonymous contributions
The methodology enables large-scale attack analysis
Abstract
The damage personal attacks cause to online discourse motivates many platforms to try to curb the phenomenon. However, understanding the prevalence and impact of personal attacks in online platforms at scale remains surprisingly difficult. The contribution of this paper is to develop and illustrate a method that combines crowdsourcing and machine learning to analyze personal attacks at scale. We show an evaluation method for a classifier in terms of the aggregated number of crowd-workers it can approximate. We apply our methodology to English Wikipedia, generating a corpus of over 100k high quality human-labeled comments and 63M machine-labeled ones from a classifier that is as good as the aggregate of 3 crowd-workers, as measured by the area under the ROC curve and Spearman correlation. Using this corpus of machine-labeled scores, our methodology allows us to explore some of the open…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsHate Speech and Cyberbullying Detection · Wikis in Education and Collaboration · Spam and Phishing Detection
