SoK: Content Moderation for End-to-End Encryption

TL;DR

This paper systematically reviews content moderation challenges and approaches in end-to-end encryption settings, covering diverse topics from malware to policy, and proposes a comprehensive process pipeline for future research.

Contribution

It synthesizes interdisciplinary literature on content moderation in E2EE, develops a process pipeline, and analyzes cryptography and policy choices across stages.

Findings

Identifies diverse content moderation challenges in E2EE environments.

Proposes a process pipeline for systematic study of E2EE content moderation.

Highlights gaps and future research directions in cryptography and policy.

Abstract

Popular messaging applications now enable end-to-end-encryption (E2EE) by default, and E2EE data storage is becoming common. These important advances for security and privacy create new content moderation challenges for online services, because services can no longer directly access plaintext content. While ongoing public policy debates about E2EE and content moderation in the United States and European Union emphasize child sexual abuse material and misinformation in messaging and storage, we identify and synthesize a wealth of scholarship that goes far beyond those topics. We bridge literature that is diverse in both content moderation subject matter, such as malware, spam, hate speech, terrorist content, and enterprise policy compliance, as well as intended deployments, including not only privacy-preserving content moderation for messaging, email, and cloud storage, but also private introspection of encrypted web traffic by middleboxes. In this work, we systematize the study of content moderation in E2EE settings. We set out a process pipeline for content moderation, drawing on a broad interdisciplinary literature that is not specific to E2EE. We examine cryptography and policy design choices at all stages of this pipeline, and we suggest areas of future research to fill gaps in literature and better understand possible paths forward.