"Model Cards for Model Reporting" in 2024: Reclassifying Category of Ethical Considerations in Terms of Trustworthiness and Risk Management

TL;DR

This paper proposes reclassifying ethical considerations in model cards into trustworthiness and risk management categories, aligning documentation with evolving trustworthy AI principles to improve transparency and evaluation.

Contribution

It introduces a new classification framework for model cards, separating trustworthiness from risk management based on recent developments in trustworthy AI guidelines.

Findings

Trustworthiness includes accountability, explainability, fairness, privacy, reliability, robustness, safety, security, transparency.

The reclassification aligns model documentation with current AI trustworthiness standards.

A two-step process for updating model card categories is proposed.

Abstract

In 2019, the paper entitled "Model Cards for Model Reporting" introduced a new tool for documenting model performance and encouraged the practice of transparent reporting for a defined list of categories. One of the categories detailed in that paper is ethical considerations, which includes the subcategories of data, human life, mitigations, risks and harms, and use cases. We propose to reclassify this category in the original model card due to the recent maturing of the field known as trustworthy AI, a term which analyzes whether the algorithmic properties of the model indicate that the AI system is deserving of trust from its stakeholders. In our examination of trustworthy AI, we highlight three respected organizations - the European Commission's High-Level Expert Group on AI, the OECD, and the U.S.-based NIST - that have written guidelines on various aspects of trustworthy AI. These recent publications converge on numerous characteristics of the term, including accountability, explainability, fairness, privacy, reliability, robustness, safety, security, and transparency, while recognizing that the implementation of trustworthy AI varies by context. Our reclassification of the original model-card category known as ethical considerations involves a two-step process: 1) adding a new category known as trustworthiness, where the subcategories will be derived from the discussion of trustworthy AI in our paper, and 2) maintaining the subcategories of ethical considerations under a renamed category known as risk environment and risk management, a title which we believe better captures today's understanding of the essence of these topics. We hope that this reclassification will further the goals of the original paper and continue to prompt those releasing trained models to accompany these models with documentation that will assist in the evaluation of their algorithmic properties.