A Multisite, Report-Based, Centralized Infrastructure for Feedback and Monitoring of Radiology AI/ML Development and Clinical Deployment

TL;DR

This paper presents a centralized, multisite infrastructure that leverages routine radiology reporting and NLP to efficiently collect, label, and monitor AI/ML models in clinical settings, reducing bias and resource use.

Contribution

It introduces an innovative, report-based system integrating image viewing, NLP, and cloud storage for continuous AI monitoring and development without burdening radiologists.

Findings

Enables multisite data collection and labeling during routine interpretation

Supports regulatory compliance for AI post-marketing surveillance

Reduces resource requirements compared to expert labeling

Abstract

An infrastructure for multisite, geographically-distributed creation and collection of diverse, high-quality, curated and labeled radiology image data is crucial for the successful automated development, deployment, monitoring and continuous improvement of Artificial Intelligence (AI)/Machine Learning (ML) solutions in the real world. An interactive radiology reporting approach that integrates image viewing, dictation, natural language processing (NLP) and creation of hyperlinks between image findings and the report, provides localized labels during routine interpretation. These images and labels can be captured and centralized in a cloud-based system. This method provides a practical and efficient mechanism with which to monitor algorithm performance. It also supplies feedback for iterative development and quality improvement of new and existing algorithmic models. Both feedback and monitoring are achieved without burdening the radiologist. The method addresses proposed regulatory requirements for post-marketing surveillance and external data. Comprehensive multi-site data collection assists in reducing bias. Resource requirements are greatly reduced compared to dedicated retrospective expert labeling.