Mapping the Stochastic Penal Colony

TL;DR

This paper reinterprets content moderation as a stochastic penal colony, analyzing unjust punishment practices through a novel methodology applied to three case studies of social media and AI platforms.

Contribution

It introduces a new conceptual model of algorithmic punishment and a methodology combining auto-ethnography and procedural justice for analysis.

Findings

Identifies patterns of unjust punishment in social media moderation.

Highlights manipulative moderation practices across platforms.

Proposes a new framework for understanding algorithmic penal systems.

Abstract

With peak content moderation seemingly behind us, this paper revisits its punitive side. But instead of focusing on who is being (disproportionately) moderated, it focuses on the punishment itself and explores the question of how content moderation treats users posting violative content unjustly, while the organizations doing the moderation act in a self-serving manner. First, this paper reworks Foucault's model of the penal system for the algorithmic age, restoring the penal colony as a figuratively liminal practice between punishment as performance and punishment as discipline, i.e., the stochastic penal colony. Second, it develops a novel methodology that combines auto-ethnography for collecting experiences and artifacts with procedural justice for analyzing them. Third, it applies this conceptual and methodological framing to three case studies, one on pre-Musk Twitter's gallingly performative moderation, one on OpenAI's exhaustively controlling moderation for DALL-E 2, and one on Pinterest's underhandedly manipulative moderation. While substantially different, all three feature the pervasive threat of account suspension, which banishes users to the stochastic penal colony.