AI Data Development: A Scorecard for the System Card Framework
Tadesse K. Bahiru, Haileleol Tibebu, and Ioannis A. Kakadiaris

TL;DR
This paper presents a comprehensive scorecard framework for evaluating AI dataset development, emphasizing transparency, quality, and ethical considerations to improve AI system reliability.
Contribution
It introduces a structured scorecard method based on the system card framework to assess and enhance AI dataset quality and transparency.
Findings
Applied to four datasets, revealing strengths and areas for improvement.
Provides tailored recommendations for dataset development and ethical practices.
Addresses both technical and ethical aspects of data quality.
Abstract
Artificial intelligence has transformed numerous industries, from healthcare to finance, enhancing decision-making through automated systems. However, the reliability of these systems is mainly dependent on the quality of the underlying datasets, raising ongoing concerns about transparency, accountability, and potential biases. This paper introduces a scorecard designed to evaluate the development of AI datasets, focusing on five key areas from the system card framework data development life cycle: data dictionary, collection process, composition, motivation, and pre-processing. The method follows a structured approach, using an intake form and scoring criteria to assess the quality and completeness of the data set. Applied to four diverse datasets, the methodology reveals strengths and improvement areas. The results are compiled using a scoring system that provides tailored…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsBig Data and Business Intelligence
