A Framework of Severity for Harmful Content Online

TL;DR

This paper introduces a comprehensive theoretical framework of severity for harmful online content, categorizing harm types and dimensions to improve research, policy, and moderation practices.

Contribution

It presents a novel, empirically grounded framework of severity for online harms, based on interviews and card-sorting with 52 participants, to aid harm understanding and prioritization.

Findings

Identified four Types of Harm: physical, emotional, relational, financial.

Defined eight Dimensions for assessing harm severity, including intent and vulnerability.

Provided application scenarios for research and policy to better address online harms.

Abstract

The proliferation of harmful content on online social media platforms has necessitated empirical understandings of experiences of harm online and the development of practices for harm mitigation. Both understandings of harm and approaches to mitigating that harm, often through content moderation, have implicitly embedded frameworks of prioritization - what forms of harm should be researched, how policy on harmful content should be implemented, and how harmful content should be moderated. To aid efforts of better understanding the variety of online harms, how they relate to one another, and how to prioritize harms relevant to research, policy, and practice, we present a theoretical framework of severity for harmful online content. By employing a grounded theory approach, we developed a framework of severity based on interviews and card-sorting activities conducted with 52 participants over the course of ten months. Through our analysis, we identified four Types of Harm (physical, emotional, relational, and financial) and eight Dimensions along which the severity of harm can be understood (perspectives, intent, agency, experience, scale, urgency, vulnerability, sphere). We describe how our framework can be applied to both research and policy settings towards deeper understandings of specific forms of harm (e.g., harassment) and prioritization frameworks when implementing policies encompassing many forms of harm.