To Act or React: Investigating Proactive Strategies For Online Community Moderation

TL;DR

This paper explores proactive moderation strategies on Reddit, using explainable machine learning to predict communities at risk of becoming hateful or dangerous, aiming to improve early intervention and reduce harmful discourse.

Contribution

It introduces a predictive framework for identifying at-risk communities before they become problematic, enhancing moderation effectiveness with explainable AI insights.

Findings

Communities evolve in ways that can be predicted months in advance.

Explainable machine learning identifies key predictors of dangerous community development.

Proactive moderation can potentially reduce the spread of hateful content.

Abstract

Reddit administrators have generally struggled to prevent or contain such discourse for several reasons including: (1) the inability for a handful of human administrators to track and react to millions of posts and comments per day and (2) fear of backlash as a consequence of administrative decisions to ban or quarantine hateful communities. Consequently, as shown in our background research, administrative actions (community bans and quarantines) are often taken in reaction to media pressure following offensive discourse within a community spilling into the real world with serious consequences. In this paper, we investigate the feasibility of proactive moderation on Reddit -- i.e., proactively identifying communities at risk of committing offenses that previously resulted in bans for other communities. Proactive moderation strategies show promise for two reasons: (1) they have potential to narrow down the communities that administrators need to monitor for hateful content and (2) they give administrators a scientific rationale to back their administrative decisions and interventions. Our work shows that communities are constantly evolving in their user base and topics of discourse and that evolution into hateful or dangerous (i.e., considered bannable by Reddit administrators) communities can often be predicted months ahead of time. This makes proactive moderation feasible. Further, we leverage explainable machine learning to help identify the strongest predictors of evolution into dangerous communities. This provides administrators with insights into the characteristics of communities at risk becoming dangerous or hateful. Finally, we investigate, at scale, the impact of participation in hateful and dangerous subreddits and the effectiveness of community bans and quarantines on the behavior of members of these communities.