Why AI Harms Can't Be Fixed One Identity at a Time: What 5300 Incident Reports Reveal About Intersectionality

TL;DR

This study analyzes 5,300 AI incident reports to reveal that intersectional harms are widespread and often amplified at specific social identity intersections, highlighting the need for intersectionality in AI risk assessment.

Contribution

It introduces a large-scale analysis of documented AI incidents using LLMs, demonstrating the importance of considering multiple social identities simultaneously in harm assessment.

Findings

Age and political identity are as common as race and gender in AI harms.

Harm amplification occurs up to three times at specific intersections.

Intersectional harms are underrepresented in current AI risk assessments.

Abstract

AI risk assessment is the primary tool for identifying harms caused by AI systems. These include intersectional harms, which arise from the interaction between identity categories (e.g., class and skin tone) and which do not occur, or occur differently, when those categories are considered separately. Yet existing AI risk assessments are still built around isolated identity categories, and when intersections are considered, they focus almost exclusively on race and gender. Drawing on a large-scale analysis of documented AI incidents, we show that AI harms do not occur one identity category at a time. Using a structured rubric applied with a Large Language Model (LLM), we analyze 5,300 reports from 1,200 documented incidents in the AI Incident Database, the most curated source of incident data. From these reports, we identify 1,513 harmed subjects and their associated identity categories, achieving 98% accuracy. At the level of individual categories, we find that age and political identity appear in documented AI harms at rates comparable to race and gender. At the level of intersecting categories, harm is amplified up to three times at specific intersections: adolescent girls, lower-class people of color, and upper-class political elites. We argue that intersectionality should be a core component of AI risk assessment to more accurately capture how harms are produced and distributed across social groups.