Everywhere Valid Bounds on False Discovery Proportions in Conformal Inference

TL;DR

This paper develops finite-sample, distribution-free bounds on the false discovery proportion in conformal inference, allowing for flexible, high-probability guarantees even after data-driven threshold selection.

Contribution

It introduces a novel method to obtain simultaneous, high-probability FDP bounds that are valid regardless of threshold choice and data inspection, improving over existing expected-value controls.

Findings

Bounds are valid uniformly over all thresholds.

Bounds are less conservative than previous methods.

Framework applies to outlier detection and conformal selection.

Abstract

Modern applications of conformal inference to multiple testing problems, such as outlier detection and candidate selection, often involve selecting test samples whose conformal p-values fall below a threshold. The quality of such methods is often measured by the false discovery proportion (FDP), defined as the fraction of incorrect selections. Existing approaches typically control the expected value of the FDP, using methods such as the Benjamini-Hochberg procedure. This approach fails to provide high-probability bounds on the realized false discovery proportion and invalidates statistical guarantees if the rejection threshold is selected after inspecting the data. This paper establishes finite-sample, distribution-free upper bounds on the FDP that hold simultaneously over all possible rejection thresholds, enabling arbitrary post hoc selection of the threshold. Simultaneous validity is achieved by constructing a high-probability envelope for the empirical distribution function of null conformal p-values by sampling from their joint distribution. Furthermore, our framework allows practitioners to modulate the envelope's shape, thereby producing tight bounds in rejection regions of primary interest. We use this flexible approach to derive simultaneous FDP upper bounds for both outlier detection and conformal selection. We demonstrate through synthetic and real-data experiments that the resulting bounds are both valid and substantially less conservative than those derived from existing approaches.