Scalable Drift Monitoring in Medical Imaging AI

TL;DR

This paper introduces MMC+, an advanced scalable framework for real-time drift detection in medical imaging AI, enhancing reliability and early warning capabilities in clinical environments.

Contribution

MMC+ extends the CheXstray framework with improved scalability, robustness, and uncertainty bounds, enabling effective drift monitoring across diverse data streams in healthcare.

Findings

Successfully detects data shifts during COVID-19 at Massachusetts General Hospital.

Correlates data drift with model performance changes.

Provides a cost-effective alternative to continuous monitoring.

Abstract

The integration of artificial intelligence (AI) into medical imaging has advanced clinical diagnostics but poses challenges in managing model drift and ensuring long-term reliability. To address these challenges, we develop MMC+, an enhanced framework for scalable drift monitoring, building upon the CheXstray framework that introduced real-time drift detection for medical imaging AI models using multi-modal data concordance. This work extends the original framework's methodologies, providing a more scalable and adaptable solution for real-world healthcare settings and offers a reliable and cost-effective alternative to continuous performance monitoring addressing limitations of both continuous and periodic monitoring methods. MMC+ introduces critical improvements to the original framework, including more robust handling of diverse data streams, improved scalability with the integration of foundation models like MedImageInsight for high-dimensional image embeddings without site-specific training, and the introduction of uncertainty bounds to better capture drift in dynamic clinical environments. Validated with real-world data from Massachusetts General Hospital during the COVID-19 pandemic, MMC+ effectively detects significant data shifts and correlates them with model performance changes. While not directly predicting performance degradation, MMC+ serves as an early warning system, indicating when AI systems may deviate from acceptable performance bounds and enabling timely interventions. By emphasizing the importance of monitoring diverse data streams and evaluating data shifts alongside model performance, this work contributes to the broader adoption and integration of AI solutions in clinical settings.