Definition drives design: Disability models and mechanisms of bias in AI technologies

TL;DR

This paper explores how different definitions of disability influence AI design choices, leading to biases, and emphasizes the importance of inclusive, transparent processes to create fairer AI systems for disabled people.

Contribution

It introduces a framework linking disability models to AI design decisions, highlighting how definitions of disability impact bias and advocating for participatory, disability-led AI development.

Findings

Disability definitions significantly influence AI problem formulation and data selection.

Design decisions based on disability models can lead to varied biases in AI outcomes.

Transparency and disabled participation mitigate bias and promote fairness.

Abstract

The increasing deployment of artificial intelligence (AI) tools to inform decision making across diverse areas including healthcare, employment, social benefits, and government policy, presents a serious risk for disabled people, who have been shown to face bias in AI implementations. While there has been significant work on analysing and mitigating algorithmic bias, the broader mechanisms of how bias emerges in AI applications are not well understood, hampering efforts to address bias where it begins. In this article, we illustrate how bias in AI-assisted decision making can arise from a range of specific design decisions, each of which may seem self-contained and non-biasing when considered separately. These design decisions include basic problem formulation, the data chosen for analysis, the use the AI technology is put to, and operational design elements in addition to the core algorithmic design. We draw on three historical models of disability common to different decision-making settings to demonstrate how differences in the definition of disability can lead to highly distinct decisions on each of these aspects of design, leading in turn to AI technologies with a variety of biases and downstream effects. We further show that the potential harms arising from inappropriate definitions of disability in fundamental design stages are further amplified by a lack of transparency and disabled participation throughout the AI design process. Our analysis provides a framework for critically examining AI technologies in decision-making contexts and guiding the development of a design praxis for disability-related AI analytics. We put forth this article to provide key questions to facilitate disability-led design and participatory development to produce more fair and equitable AI technologies in disability-related contexts.