The "Who", "What", and "How" of Responsible AI Governance: A Systematic Review and Meta-Analysis of (Actor, Stage)-Specific Tools

TL;DR

This paper systematically reviews and analyzes over 220 responsible AI tools, revealing significant gaps in coverage across stakeholder roles and lifecycle stages, highlighting areas for improvement in AI governance tools and practices.

Contribution

It categorizes existing tools by AI actors and stages, identifying imbalances and gaps in responsible AI support, and emphasizes the need for more holistic and validated tools.

Findings

Most tools support AI designers during data and modeling stages

Neglect of roles like leadership, deployers, end-users, and stages like deployment

Few tools are validated for usability and effectiveness

Abstract

The implementation of responsible AI in an organization is inherently complex due to the involvement of multiple stakeholders, each with their unique set of goals and responsibilities across the entire AI lifecycle. These responsibilities are often ambiguously defined and assigned, leading to confusion, miscommunication, and inefficiencies. Even when responsibilities are clearly defined and assigned to specific roles, the corresponding AI actors lack effective tools to support their execution. Toward closing these gaps, we present a systematic review and comprehensive meta-analysis of the current state of responsible AI tools, focusing on their alignment with specific stakeholder roles and their responsibilities in various AI lifecycle stages. We categorize over 220 tools according to AI actors and stages they address. Our findings reveal significant imbalances across the stakeholder roles and lifecycle stages addressed. The vast majority of available tools have been created to support AI designers and developers specifically during data-centric and statistical modeling stages while neglecting other roles such as institutional leadership, deployers, end-users, and impacted communities, and stages such as value proposition and deployment. The uneven distribution we describe here highlights critical gaps that currently exist in responsible AI governance research and practice. Our analysis reveals that despite the myriad of frameworks and tools for responsible AI, it remains unclear \emph{who} within an organization and \emph{when} in the AI lifecycle a tool applies. Furthermore, existing tools are rarely validated, leaving critical gaps in their usability and effectiveness. These gaps provide a starting point for researchers and practitioners to create more effective and holistic approaches to responsible AI development and governance.