A Methodology for Creating AI FactSheets

TL;DR

This paper introduces a methodology for creating AI FactSheets, a form of documentation that enhances transparency and trust in AI models, especially in high-stakes applications.

Contribution

It is the first to describe a systematic methodology for developing AI FactSheets, including practical insights from creating nearly two dozen such documents.

Findings

The methodology covers key issues and questions for organizations.

Creating FactSheets improves transparency and trust.

Insights facilitate broader adoption of AI documentation practices.

Abstract

As AI models and services are used in a growing number of highstakes areas, a consensus is forming around the need for a clearer record of how these models and services are developed to increase trust. Several proposals for higher quality and more consistent AI documentation have emerged to address ethical and legal concerns and general social impacts of such systems. However, there is little published work on how to create this documentation. This is the first work to describe a methodology for creating the form of AI documentation we call FactSheets. We have used this methodology to create useful FactSheets for nearly two dozen models. This paper describes this methodology and shares the insights we have gathered. Within each step of the methodology, we describe the issues to consider and the questions to explore with the relevant people in an organization who will be creating and consuming the AI facts in a FactSheet. This methodology will accelerate the broader adoption of transparent AI documentation.