A theory of ROC analysis of rule-out and rule-in diagnostics with applications to mammography data

TL;DR

This paper develops a copula-based theoretical framework to analyze ROC curves for correlated diagnostic tests, applied to mammography data with AI assistance, revealing how correlations affect diagnostic performance.

Contribution

It introduces a novel copula-based theory for ROC analysis of correlated tests and applies it to mammography data with AI, providing insights into test correlation effects.

Findings

Increasing radiologist-AI correlation for diseased cases improves AUC in rule-out scenarios.

Decreasing correlation for non-diseased cases enhances rule-out ROC performance.

Empirical data aligns with the theoretical predictions about correlation effects.

Abstract

Multiple diagnostic tests are frequently used to determine the presence of a disease condition in patients. In this paper, we use bivariate copulas to examine the properties of receiver operating characteristic (ROC) curves formed when two correlated diagnostic tests are used together to rule-out ("believe the negative") and rule-in ("believe the positive") patients for disease. We use this theory to analyze three mammography data sets where AI devices are applied to reduce radiologists' workload or improve diagnostic performance. Our analysis shows with generality that increasing the radiologist-AI correlation for diseased cases enhances the area under the ROC curve (AUC) of a radiologist-AI rule-out curve, whereas decreasing correlation for non-diseased cases has a similar effect. The opposite trends hold for rule-in scenarios. Applications to clinical mammography data show that projected empirical radiologist performance under a rule-out or rule-in scenario is consistent with the theory.