Towards Quality Management of Machine Learning Systems for Medical Applications

TL;DR

This paper discusses the challenges of integrating machine learning systems into clinical practice, emphasizing the need for quality management, robustness assessment, and risk evaluation tailored for medical applications.

Contribution

It introduces a framework based on AAPM TG 100 guidelines for developing quality management systems for ML in healthcare, considering interpretability and robustness.

Findings

Robustness metrics alone are insufficient for clinical reliability.

A risk-based framework can incorporate ML system quality considerations.

Interpretability impacts risk assessment and quality management development.

Abstract

The use of machine learning systems in clinical routine is still hampered by the necessity of a medical device certification and/or by difficulty to implement these systems in a clinic's quality management system. In this context, the key questions for a user are how to ensure reliable model predictions and how to appraise the quality of a model's results on a regular basis. In this paper we first review why the common out-of-sample performance metrics are not sufficient for assessing the robustness of model predictions. We discuss some conceptual foundation for a clinical implementation of a machine learning system and argue that both vendors and users should take certain responsibilities, as is already common practice for high-risk medical equipment. Along this line the best practices for dealing with robustness (or absence thereof) of machine learning models are revisited. We propose the methodology from AAPM Task Group 100 report no. 283 as a natural framework for developing a quality management program for a clinical process that encompasses a machine learning system. This is illustrated with an explicit albeit generic example. Our analysis shows how the risk evaluation in this framework can accommodate machine learning systems independently of their robustness evaluation. In particular, we highlight how the degree of interpretability of a machine learning system can be accounted for systematically within the risk evaluation and in the development of a quality management system.