Content Removal as a Moderation Strategy: Compliance and Other Outcomes in the ChangeMyView Community

TL;DR

This study investigates whether comment removal in online communities like Reddit's ChangeMyView subreddit causally improves user behavior, finding it reduces immediate noncompliance but does not necessarily induce broader behavioral improvements.

Contribution

It introduces a novel 'delayed feedback' causal inference method to distinguish the effects of comment removal from mere non-compliance, advancing understanding of moderation impacts.

Findings

Comment deletion causally reduces immediate noncompliance.

No evidence found that deletion induces broader behavioral change.

Highlights limits of content removal as a moderation tool.

Abstract

Moderators of online communities often employ comment deletion as a tool. We ask here whether, beyond the positive effects of shielding a community from undesirable content, does comment removal actually cause the behavior of the comment's author to improve? We examine this question in a particularly well-moderated community, the ChangeMyView subreddit. The standard analytic approach of interrupted time-series analysis unfortunately cannot answer this question of causality because it fails to distinguish the effect of having made a non-compliant comment from the effect of being subjected to moderator removal of that comment. We therefore leverage a "delayed feedback" approach based on the observation that some users may remain active between the time when they posted the non-compliant comment and the time when that comment is deleted. Applying this approach to such users, we reveal the causal role of comment deletion in reducing immediate noncompliance rates, although we do not find evidence of it having a causal role in inducing other behavior improvements. Our work thus empirically demonstrates both the promise and some potential limits of content removal as a positive moderation strategy, and points to future directions for identifying causal effects from observational data.