The METRIC-framework for assessing data quality for trustworthy AI in medicine: a systematic review

TL;DR

This paper systematically reviews data quality assessment methods in medical AI, proposing the METRIC-framework with 15 dimensions to improve trustworthiness, robustness, and regulatory approval of machine learning models in healthcare.

Contribution

It introduces the METRIC-framework, a novel data quality assessment tool specifically designed for medical machine learning datasets, integrating existing knowledge and addressing trustworthiness concerns.

Findings

Identified 62 relevant studies on data quality in medical AI.

Proposed the METRIC-framework with 15 dimensions for dataset assessment.

Highlighted the importance of data quality for regulatory approval and trustworthy AI.

Abstract

The adoption of machine learning (ML) and, more specifically, deep learning (DL) applications into all major areas of our lives is underway. The development of trustworthy AI is especially important in medicine due to the large implications for patients' lives. While trustworthiness concerns various aspects including ethical, technical and privacy requirements, we focus on the importance of data quality (training/test) in DL. Since data quality dictates the behaviour of ML products, evaluating data quality will play a key part in the regulatory approval of medical AI products. We perform a systematic review following PRISMA guidelines using the databases PubMed and ACM Digital Library. We identify 2362 studies, out of which 62 records fulfil our eligibility criteria. From this literature, we synthesise the existing knowledge on data quality frameworks and combine it with the perspective of ML applications in medicine. As a result, we propose the METRIC-framework, a specialised data quality framework for medical training data comprising 15 awareness dimensions, along which developers of medical ML applications should investigate a dataset. This knowledge helps to reduce biases as a major source of unfairness, increase robustness, facilitate interpretability and thus lays the foundation for trustworthy AI in medicine. Incorporating such systematic assessment of medical datasets into regulatory approval processes has the potential to accelerate the approval of ML products and builds the basis for new standards.