"Over-the-Hood" AI Inclusivity Bugs and How 3 AI Product Teams Found and Fixed Them

TL;DR

This study investigates 'over-the-hood' inclusivity biases in user-facing AI products, revealing common bug types, effective fixes, and a tailored inclusive design method to improve AI fairness for diverse users.

Contribution

It introduces a field study identifying AI inclusivity bugs, and develops a specialized version of the GenderMag method for detecting such biases.

Findings

Identified 6 types of AI inclusivity bugs occurring 83 times

Implemented fixes for 47 bug instances

Developed GenderMag-for-AI, an effective bias detection tool

Abstract

While much research has shown the presence of AI's "under-the-hood" biases (e.g., algorithmic, training data, etc.), what about "over-the-hood" inclusivity biases: barriers in user-facing AI products that disproportionately exclude users with certain problem-solving approaches? Recent research has begun to report the existence of such biases -- but what do they look like, how prevalent are they, and how can developers find and fix them? To find out, we conducted a field study with 3 AI product teams, to investigate what kinds of AI inclusivity bugs exist uniquely in user-facing AI products, and whether/how AI product teams might harness an existing (non-AI-oriented) inclusive design method to find and fix them. The teams' work resulted in identifying 6 types of AI inclusivity bugs arising 83 times, fixes covering 47 of these bug instances, and a new variation of the GenderMag inclusive design method, GenderMag-for-AI, that is especially effective at detecting certain kinds of AI inclusivity bugs.