Disability Across Cultures: A Human-Centered Audit of Ableism in Western and Indic LLMs

TL;DR

This study compares Western and Indic language models in recognizing ableist language, revealing significant biases and highlighting the importance of local context in AI systems for disability rights.

Contribution

It introduces a cross-cultural evaluation of LLMs' ability to detect ableism, emphasizing the need for localized models and inclusive standards in AI.

Findings

Western LLMs overestimate ableist harm

Indic LLMs underestimate ableist harm

Models are more tolerant of ableism in Hindi and Western framings

Abstract

People with disabilities (PwD) experience disproportionately high levels of discrimination and hate online, particularly in India, where entrenched stigma and limited resources intensify these challenges. Large language models (LLMs) are increasingly used to identify and mitigate online hate, yet most research on online ableism focuses on Western audiences with Western AI models. Are these models adequately equipped to recognize ableist harm in non-Western places like India? Do localized, Indic language models perform better? To investigate, we adopted and translated a publicly available ableist speech dataset to Hindi, and prompted eight LLMs--four developed in the U.S. (GPT-4, Gemini, Claude, Llama) and four in India (Krutrim, Nanda, Gajendra, Airavata)--to score and explain ableism. In parallel, we recruited 175 PwD from both the U.S. and India to perform the same task, revealing stark differences between groups. Western LLMs consistently overestimated ableist harm, while Indic LLMs underestimated it. Even more concerning, all LLMs were more tolerant of ableism when it was expressed in Hindi and asserted Western framings of ableist harm. In contrast, Indian PwD interpreted harm through intention, relationality, and resilience--emphasizing a desire to inform and educate perpetrators. This work provides groundwork for global, inclusive standards of ableism, demonstrating the need to center local disability experiences in the design and evaluation of AI systems.