Datasheets for AI and medical datasets (DAIMS): a data validation and documentation framework before machine learning analysis in medical research

TL;DR

DAIMS extends existing data documentation frameworks by providing a comprehensive checklist, software tools, and flowcharts to standardize and validate medical datasets for machine learning research.

Contribution

The paper introduces DAIMS, a new framework with tools and guidelines specifically designed for preparing and documenting medical datasets for ML applications.

Findings

DAIMS includes a 24-item checklist for data standardization.

The framework offers a software tool for data validation.

DAIMS provides a flowchart linking research questions to ML methods.

Abstract

Despite progresses in data engineering, there are areas with limited consistencies across data validation and documentation procedures causing confusions and technical problems in research involving machine learning. There have been progresses by introducing frameworks like "Datasheets for Datasets", however there are areas for improvements to prepare datasets, ready for ML pipelines. Here, we extend the framework to "Datasheets for AI and medical datasets - DAIMS." Our publicly available solution, DAIMS, provides a checklist including data standardization requirements, a software tool to assist the process of the data preparation, an extended form for data documentation and pose research questions, a table as data dictionary, and a flowchart to suggest ML analyses to address the research questions. The checklist consists of 24 common data standardization requirements, where the tool checks and validate a subset of them. In addition, we provided a flowchart mapping research questions to suggested ML methods. DAIMS can serve as a reference for standardizing datasets and a roadmap for researchers aiming to apply effective ML techniques in their medical research endeavors. DAIMS is available on GitHub and as an online app to automate key aspects of dataset evaluation, facilitating efficient preparation of datasets for ML studies.