Social Turing Tests: Crowdsourcing Sybil Detection

TL;DR

This paper investigates the potential of crowdsourcing human judgment to detect Sybil accounts in online social networks, demonstrating that expert human detection can be highly effective and scalable as a detection method.

Contribution

It introduces a novel crowdsourced Sybil detection system based on human judgment, validated through extensive user studies on real-world social network data.

Findings

Experts achieve near-optimal detection accuracy.

Crowdsourcing can be scaled for large networks.

The system complements existing automated detection tools.

Abstract

As popular tools for spreading spam and malware, Sybils (or fake accounts) pose a serious threat to online communities such as Online Social Networks (OSNs). Today, sophisticated attackers are creating realistic Sybils that effectively befriend legitimate users, rendering most automated Sybil detection techniques ineffective. In this paper, we explore the feasibility of a crowdsourced Sybil detection system for OSNs. We conduct a large user study on the ability of humans to detect today's Sybil accounts, using a large corpus of ground-truth Sybil accounts from the Facebook and Renren networks. We analyze detection accuracy by both "experts" and "turkers" under a variety of conditions, and find that while turkers vary significantly in their effectiveness, experts consistently produce near-optimal results. We use these results to drive the design of a multi-tier crowdsourcing Sybil detection system. Using our user study data, we show that this system is scalable, and can be highly effective either as a standalone system or as a complementary technique to current tools.