AI Data Development: A Scorecard for the System Card Framework

TL;DR

This paper presents a comprehensive scorecard framework for evaluating AI dataset development, emphasizing transparency, quality, and ethical considerations to improve AI system reliability.

Contribution

It introduces a structured scorecard method based on the system card framework to assess and enhance AI dataset quality and transparency.

Findings

Applied to four datasets, revealing strengths and areas for improvement.

Provides tailored recommendations for dataset development and ethical practices.

Addresses both technical and ethical aspects of data quality.

Abstract

Artificial intelligence has transformed numerous industries, from healthcare to finance, enhancing decision-making through automated systems. However, the reliability of these systems is mainly dependent on the quality of the underlying datasets, raising ongoing concerns about transparency, accountability, and potential biases. This paper introduces a scorecard designed to evaluate the development of AI datasets, focusing on five key areas from the system card framework data development life cycle: data dictionary, collection process, composition, motivation, and pre-processing. The method follows a structured approach, using an intake form and scoring criteria to assess the quality and completeness of the data set. Applied to four diverse datasets, the methodology reveals strengths and improvement areas. The results are compiled using a scoring system that provides tailored recommendations to enhance the transparency and integrity of the data set. The scorecard addresses technical and ethical aspects, offering a holistic evaluation of data practices. This approach aims to improve the quality of the data set. It offers practical guidance to curators and researchers in developing responsible AI systems, ensuring fairness and accountability in decision support systems.