"Cold, Calculated, and Condescending": How AI Identifies and Explains Ableism Compared to Disabled People

TL;DR

This study evaluates how well AI models detect and explain ableism in social media comments compared to disabled people's judgments, revealing significant gaps in accuracy, nuance, and understanding.

Contribution

It introduces a novel dataset of ableist comments, compares AI and disabled people's assessments, and critically analyzes AI explanations, highlighting areas for improvement in moderation systems.

Findings

AI underestimates toxicity compared to disabled people's ratings.

AI explanations lack nuance and sometimes make incorrect assumptions.

AI models show inconsistent ability to identify and explain ableism.

Abstract

People with disabilities (PwD) regularly encounter ableist hate and microaggressions online. These spaces are generally moderated by machine learning models, but little is known about how effectively AI models identify ableist speech and how well their judgments align with PwD. To investigate this, we curated a first-of-its-kind dataset of 200 social media comments targeted towards PwD, and prompted state-of-the art AI models (i.e., Toxicity Classifiers, LLMs) to score toxicity and ableism for each comment, and explain their reasoning. Then, we recruited 190 participants to similarly rate and explain the harm, and evaluate LLM explanations. Our mixed-methods analysis highlighted a major disconnect: AI underestimated toxicity compared to PwD ratings, while its ableism assessments were sporadic and varied. Although LLMs identified some biases, its explanations were flawed--they lacked nuance, made incorrect assumptions, and appeared judgmental instead of educational. Going forward, we discuss challenges and opportunities in designing moderation systems for ableism, and advocate for the involvement of intersectional disabled perspectives in AI.