VEWS: A Wikipedia Vandal Early Warning System

TL;DR

This paper introduces VEWS, an early vandal detection system for Wikipedia that uses novel features and machine learning approaches to identify vandals with high accuracy before traditional methods do.

Contribution

The paper develops three innovative feature sets and combines them into VEWS, achieving over 85% accuracy in vandal detection without relying on user-reported reverts.

Findings

VEWS outperforms ClueBot NG and STiki in accuracy.

VEWS detects vandals on average 2.39 edits earlier than ClueBot NG.

Combining VEWS with ClueBot NG yields an even more accurate early warning system.

Abstract

We study the problem of detecting vandals on Wikipedia before any human or known vandalism detection system reports flagging potential vandals so that such users can be presented early to Wikipedia administrators. We leverage multiple classical ML approaches, but develop 3 novel sets of features. Our Wikipedia Vandal Behavior (WVB) approach uses a novel set of user editing patterns as features to classify some users as vandals. Our Wikipedia Transition Probability Matrix (WTPM) approach uses a set of features derived from a transition probability matrix and then reduces it via a neural net auto-encoder to classify some users as vandals. The VEWS approach merges the previous two approaches. Without using any information (e.g. reverts) provided by other users, these algorithms each have over 85% classification accuracy. Moreover, when temporal recency is considered, accuracy goes to almost 90%. We carry out detailed experiments on a new data set we have created consisting of about 33K Wikipedia users (including both a black list and a white list of editors) and containing 770K edits. We describe specific behaviors that distinguish between vandals and non-vandals. We show that VEWS beats ClueBot NG and STiki, the best known algorithms today for vandalism detection. Moreover, VEWS detects far more vandals than ClueBot NG and on average, detects them 2.39 edits before ClueBot NG when both detect the vandal. However, we show that the combination of VEWS and ClueBot NG can give a fully automated vandal early warning system with even higher accuracy.