Conformal Risk Control for Pulmonary Nodule Detection

TL;DR

This paper applies conformal risk control to pulmonary nodule detection, providing statistically guaranteed prediction sets that improve decision support in lung cancer screening by balancing sensitivity and false positives.

Contribution

It introduces the use of conformal risk control for uncertainty quantification in pulmonary nodule detection, offering formal guarantees and practical insights for healthcare AI models.

Findings

Prediction sets with conformal guarantees ensure reliable uncertainty quantification.

The model achieves sensitivity comparable to radiologists, with a slight increase in false positives.

Using off-the-shelf models without accounting for ontological uncertainty can lead to risks.

Abstract

Quantitative tools are increasingly appealing for decision support in healthcare, driven by the growing capabilities of advanced AI systems. However, understanding the predictive uncertainties surrounding a tool's output is crucial for decision-makers to ensure reliable and transparent decisions. In this paper, we present a case study on pulmonary nodule detection for lung cancer screening, enhancing an advanced detection model with an uncertainty quantification technique called conformal risk control (CRC). We demonstrate that prediction sets with conformal guarantees are attractive measures of predictive uncertainty in the safety-critical healthcare domain, allowing end-users to achieve arbitrary validity by trading off false positives and providing formal statistical guarantees on model performance. Among ground-truth nodules annotated by at least three radiologists, our model achieves a sensitivity that is competitive with that generally achieved by individual radiologists, with a slight increase in false positives. Furthermore, we illustrate the risks of using off-the-shelve prediction models when faced with ontological uncertainty, such as when radiologists disagree on what constitutes the ground truth on pulmonary nodules.