Integrating Explainable AI in Medical Devices: Technical, Clinical and Regulatory Insights and Recommendations

TL;DR

This paper explores the integration of explainable AI into medical devices, emphasizing safety, trust, and regulatory considerations through expert insights, clinical evaluations, and recommendations for effective adoption in healthcare.

Contribution

It provides a comprehensive analysis of technical, clinical, and regulatory aspects of explainable AI in medical devices, including expert insights and practical recommendations.

Findings

Expert group evaluations of AI algorithm outputs in clinical decision-making

Insights into clinician-AI interaction during diagnosis

Recommendations for safe and effective AI integration in healthcare

Abstract

There is a growing demand for the use of Artificial Intelligence (AI) and Machine Learning (ML) in healthcare, particularly as clinical decision support systems to assist medical professionals. However, the complexity of many of these models, often referred to as black box models, raises concerns about their safe integration into clinical settings as it is difficult to understand how they arrived at their predictions. This paper discusses insights and recommendations derived from an expert working group convened by the UK Medicine and Healthcare products Regulatory Agency (MHRA). The group consisted of healthcare professionals, regulators, and data scientists, with a primary focus on evaluating the outputs from different AI algorithms in clinical decision-making contexts. Additionally, the group evaluated findings from a pilot study investigating clinicians' behaviour and interaction with AI methods during clinical diagnosis. Incorporating AI methods is crucial for ensuring the safety and trustworthiness of medical AI devices in clinical settings. Adequate training for stakeholders is essential to address potential issues, and further insights and recommendations for safely adopting AI systems in healthcare settings are provided.