Critical Perspectives: A Benchmark Revealing Pitfalls in PerspectiveAPI

TL;DR

This paper introduces SASS, a new benchmark to evaluate toxicity detection tools like PerspectiveAPI, revealing significant shortcomings and emphasizing the need to critically assess such tools to prevent harms.

Contribution

We propose SASS, a novel benchmark for evaluating toxicity detection models, and demonstrate that PerspectiveAPI has notable limitations on this new challenging dataset.

Findings

PerspectiveAPI shows shortcomings on SASS in multiple toxicity categories

SASS uncovers previously undetected toxic language

Evaluation of GPT-3 prompts reveals performance gaps

Abstract

Detecting "toxic" language in internet content is a pressing social and technical challenge. In this work, we focus on PERSPECTIVE from Jigsaw, a state-of-the-art tool that promises to score the "toxicity" of text, with a recent model update that claims impressive results (Lees et al., 2022). We seek to challenge certain normative claims about toxic language by proposing a new benchmark, Selected Adversarial SemanticS, or SASS. We evaluate PERSPECTIVE on SASS, and compare to low-effort alternatives, like zero-shot and few-shot GPT-3 prompt models, in binary classification settings. We find that PERSPECTIVE exhibits troubling shortcomings across a number of our toxicity categories. SASS provides a new tool for evaluating performance on previously undetected toxic language that avoids common normative pitfalls. Our work leads us to emphasize the importance of questioning assumptions made by tools already in deployment for toxicity detection in order to anticipate and prevent disparate harms.