Centering the Margins: Outlier-Based Identification of Harmed Populations in Toxicity Detection

TL;DR

This paper introduces an outlier-based method to identify marginalized groups in toxicity detection, revealing that models perform worse on these groups and that this approach uncovers broader and more intersectional harms than traditional demographic analysis.

Contribution

It operationalizes the concept of 'margins' using outlier detection to better identify vulnerable populations and their specific harms in toxicity detection models.

Findings

Model performance is up to 70.4% worse for demographic outliers.

Text outliers experience up to 68.4% higher MSE in toxicity detection.

Outlier analysis reveals broader and more intersectional harms than traditional methods.

Abstract

The impact of AI models on marginalized communities has traditionally been measured by identifying performance differences between specified demographic subgroups. Though this approach aims to center vulnerable groups, it risks obscuring patterns of harm faced by intersectional subgroups or shared across multiple groups. To address this, we draw on theories of marginalization from disability studies and related disciplines, which state that people farther from the norm face greater adversity, to consider the "margins" in the domain of toxicity detection. We operationalize the "margins" of a dataset by employing outlier detection to identify text about people with demographic attributes distant from the "norm". We find that model performance is consistently worse for demographic outliers, with mean squared error (MSE) between outliers and non-outliers up to 70.4% worse across toxicity types. It is also worse for text outliers, with a MSE up to 68.4% higher for outliers than non-outliers. We also find text and demographic outliers to be particularly susceptible to errors in the classification of severe toxicity and identity attacks. Compared to analysis of disparities using traditional demographic breakdowns, we find that our outlier analysis frequently surfaces greater harms faced by a larger, more intersectional group, which suggests that outlier analysis is particularly beneficial for identifying harms against those groups.