Testing of Deep Learning Model in Real World Clinical Setting: A Case Study in Obstetric Ultrasound

TL;DR

This paper presents a framework for deploying and evaluating AI models in real-world clinical obstetric ultrasound settings, highlighting practical challenges and user feedback for model refinement.

Contribution

It introduces a generic deployment framework and demonstrates its application in real-time fetal ultrasound, emphasizing the importance of real-world validation.

Findings

Model shows potential benefits for practitioners

Navigational guidance is a key area for improvement

Early deployment provides valuable user feedback

Abstract

Despite the rapid development of AI models in medical image analysis, their validation in real-world clinical settings remains limited. To address this, we introduce a generic framework designed for deploying image-based AI models in such settings. Using this framework, we deployed a trained model for fetal ultrasound standard plane detection, and evaluated it in real-time sessions with both novice and expert users. Feedback from these sessions revealed that while the model offers potential benefits to medical practitioners, the need for navigational guidance was identified as a key area for improvement. These findings underscore the importance of early deployment of AI models in real-world settings, leading to insights that can guide the refinement of the model and system based on actual user feedback.