Establishing Rigorous and Cost-effective Clinical Trials for Artificial Intelligence Models

TL;DR

This paper introduces rigorous, cost-effective evaluation methodologies for AI in clinical practice, including dual-centered randomized controlled trials and virtual in-silico trials, demonstrated with real medical data and clinician comparisons.

Contribution

It pioneers the use of dual-centered AI RCTs and virtual clinician-based in-silico trials as effective proxies for clinical evaluation of AI models in medicine.

Findings

VC-MedAI performs comparably to human clinicians.

DC-AI RCTs are necessary for rigorous evaluation.

Proposed methods enable fast, cost-effective AI assessment.

Abstract

A profound gap persists between artificial intelligence (AI) and clinical practice in medicine, primarily due to the lack of rigorous and cost-effective evaluation methodologies. State-of-the-art and state-of-the-practice AI model evaluations are limited to laboratory studies on medical datasets or direct clinical trials with no or solely patient-centered controls. Moreover, the crucial role of clinicians in collaborating with AI, pivotal for determining its impact on clinical practice, is often overlooked. For the first time, we emphasize the critical necessity for rigorous and cost-effective evaluation methodologies for AI models in clinical practice, featuring patient/clinician-centered (dual-centered) AI randomized controlled trials (DC-AI RCTs) and virtual clinician-based in-silico trials (VC-MedAI) as an effective proxy for DC-AI RCTs. Leveraging 7500 diagnosis records from two-step inaugural DC-AI RCTs across 14 medical centers with 125 clinicians, our results demonstrate the necessity of DC-AI RCTs and the effectiveness of VC-MedAI. Notably, VC-MedAI performs comparably to human clinicians, replicating insights and conclusions from prospective DC-AI RCTs. We envision DC-AI RCTs and VC-MedAI as pivotal advancements, presenting innovative and transformative evaluation methodologies for AI models in clinical practice, offering a preclinical-like setting mirroring conventional medicine, and reshaping development paradigms in a cost-effective and fast-iterative manner. Chinese Clinical Trial Registration: ChiCTR2400086816.