Model Cards Revisited: Bridging the Gap Between Theory and Practice for Ethical AI Requirements

TL;DR

This paper analyzes existing AI model documentation practices, identifies gaps in ethical coverage, and proposes a taxonomy to improve model cards for better ethical compliance.

Contribution

It provides a comprehensive taxonomy of 43 ethical requirements for AI model documentation, bridging the gap between theory and practice.

Findings

Developed a taxonomy with 43 ethical requirements

Found that current model cards focus mainly on capabilities and reliability

Highlighted the need for broader ethical considerations in documentation

Abstract

Model cards are the primary documentation framework for developers of artificial intelligence (AI) models to communicate critical information to their users. Those users are often developers themselves looking for relevant documentation to ensure that their AI systems comply with the ethical requirements of existing laws, guidelines, and standards. Recent studies indicate inadequate model documentation practices, suggesting a gap between AI requirements and current practices in model documentation. To understand this gap and provide actionable guidance to bridge it, we conducted a thematic analysis of 26 guidelines on ethics and AI, three AI documentation frameworks, three quantitative studies of model cards, and ten actual model cards. We identified a total of 43 ethical requirements relevant to model documentation and organized them into a taxonomy featuring four themes and twelve sub-themes representing ethical principles. Our findings indicate that model developers predominantly emphasize model capabilities and reliability in the documentation while overlooking other ethical aspects, such as explainability, user autonomy, and fairness. This underscores the need for enhanced support in documenting ethical AI considerations. Our taxonomy serves as a foundation for a revised model card framework that holistically addresses ethical AI requirements.