Between a ROC and a Hard Place: Using prevalence plots to understand the likely real world performance of biomarkers in the clinic

TL;DR

This paper emphasizes that the real-world effectiveness of biomarkers depends heavily on disease prevalence, and advocates using prevalence plots for better clinical performance prediction instead of relying solely on ROC curves.

Contribution

It introduces prevalence plots as a tool to better predict clinical biomarker performance, highlighting limitations of ROC curves in real-world settings.

Findings

ROC curves are prevalence-independent but may mislead clinical performance expectations.

Prevalence plots provide a more realistic assessment of biomarker utility in clinical settings.

Assays with high ROC AUC can perform poorly if disease prevalence is not considered.

Abstract

The Receiver Operating Characteristic (ROC) curve and the Area Under the Curve (AUC) of the ROC curve are widely used to compare the performance of diagnostic and prognostic assays. The ROC curve has the advantage that it is independent of disease prevalence. However, in this note we remind readers that the performance of an assay upon translation to the clinic is critically dependent upon that very same prevalence. Without an understanding of prevalence in the test population, even robust bioassays with excellent ROC characteristics may perform poorly in the clinic. Instead, simple plots of candidate assay performance as a function of prevalence rate give a more realistic understanding of the likely real-world performance and a greater understanding of the likely impact of variation in that prevalence on translational performance in the clinic.