A Question Bank to Assess AI Inclusivity: Mapping out the Journey from Diversity Errors to Inclusion Excellence

TL;DR

This paper presents a comprehensive question bank with 253 items to evaluate AI systems' inclusivity across multiple pillars, aiming to improve diversity and fairness in AI development and deployment.

Contribution

It introduces a structured, multi-source developed question bank for assessing AI inclusivity, filling a gap in existing risk assessment frameworks.

Findings

The question bank effectively evaluates AI inclusivity across diverse roles and domains.

Integrating D&I principles into AI workflows enhances fairness and responsibility.

Simulated user study validates the relevance and utility of the question bank.

Abstract

Ensuring diversity and inclusion (D&I) in artificial intelligence (AI) is crucial for mitigating biases and promoting equitable decision-making. However, existing AI risk assessment frameworks often overlook inclusivity, lacking standardized tools to measure an AI system's alignment with D&I principles. This paper introduces a structured AI inclusivity question bank, a comprehensive set of 253 questions designed to evaluate AI inclusivity across five pillars: Humans, Data, Process, System, and Governance. The development of the question bank involved an iterative, multi-source approach, incorporating insights from literature reviews, D&I guidelines, Responsible AI frameworks, and a simulated user study. The simulated evaluation, conducted with 70 AI-generated personas related to different AI jobs, assessed the question bank's relevance and effectiveness for AI inclusivity across diverse roles and application domains. The findings highlight the importance of integrating D&I principles into AI development workflows and governance structures. The question bank provides an actionable tool for researchers, practitioners, and policymakers to systematically assess and enhance the inclusivity of AI systems, paving the way for more equitable and responsible AI technologies.