Evaluating virtual-control-augmented trials for reproducing treatment effect from original RCTs

TL;DR

This paper evaluates the use of AI-generated virtual control data to augment RCTs, demonstrating potential for reducing sample sizes but highlighting risks of errors and misleading conclusions.

Contribution

It provides the first empirical analysis of virtual control augmentation in RCTs, comparing AI algorithms and assessing error risks.

Findings

Virtual controls can approximate traditional control outcomes.

Using virtual controls reduces required sample size.

Risks of errors and misleading results are significant.

Abstract

This study investigates the use of virtual patient data to augment control arms in randomised controlled trials (RCTs). Using data from the IST and IST3 trials, we simulated RCTs in which the recruitment in the control arms would stop after a fraction of the initially planned sample size, and would be completed by virtual patients generated by CTGAN and TVAE, two AI algorithms trained on the recruited control patients. In IST, the absolute risk difference(ARD) on death or dependency at 14 days was -0.012 (SE 0.014). Completing the control arm by CTGAN-generated virtual patients after the recruitment of 10% and 50% of participants, yielded an ARD of 0.004 (SE 0.014) (relative difference 133%) and -0.021 (SE 0.014) (relative difference 76%), respectively. Results were comparable with IST3 or TVAE. This is the first empirical demonstration of the risk of errors and misleading conclusions associated with generating virtual controls solely from trial data.