A Deployment Audit of Release-Side Risk in Conformal Triage under Prevalence Shift

TL;DR

This paper introduces a leakage-aware deployment audit for conformal triage systems, focusing on safety and effectiveness under prevalence shifts, with practical evaluation on a lung cancer screening pilot.

Contribution

It proposes a novel audit method that separates roles in conformal triage to evaluate safety-critical release decisions directly.

Findings

Lower review rates can be misleading after prevalence correction.

The classwise branch reveals insufficient event labels for safe release.

Applying the audit to NSCLC data shows safety-critical trade-offs.

Abstract

Conformal triage converts predictive scores into deployment actions that either release a case, flag it for urgent attention, or defer it to human review. Under prevalence shift, however, the usual summaries of marginal coverage and human-review rate can miss the safety-critical question of whether patients who truly experience the target event are released without review. To address this gap, we introduce a leakage-aware deployment audit for release-side conformal triage. It first assigns target subjects to three non-overlapping roles: prevalence correction, conformal calibration, and held-out release-safety evaluation. This separation then lets the audit evaluate release directly: how many event-positive patients are cleared without review, whether the pilot has enough event labels for calibration, and how the safety-review trade-off shifts. Applying this audit to a retrospective NSCLC pilot shows why lower review can be misleading: after prevalence correction, the pooled conformal branch lowers review by releasing more patients, some of whom are event-positive. Within the audit, the classwise branch acts as a scarcity diagnostic: the pilot has too few event labels to certify safe low-review release.