On Training-Conditional Conformal Prediction and Binomial Proportion Confidence Intervals

TL;DR

This paper critically examines the use of training-conditional conformal prediction for binomial proportion confidence intervals in safety certification, arguing that traditional methods are more appropriate for statistical guarantees.

Contribution

The paper clarifies the limitations of training-conditional conformal prediction in BPCI problems and advocates for traditional BPCI methods in safety certification contexts.

Findings

Training-conditional CP does not provide valid safety guarantees.

Traditional BPCI methods are more suitable for statistical safety certification.

Conformal Prediction may not be appropriate for BPCI problems.

Abstract

Estimating the expectation of a Bernoulli random variable based on N independent trials is a classical problem in statistics, typically addressed using Binomial Proportion Confidence Intervals (BPCI). In the control systems community, many critical tasks-such as certifying the statistical safety of dynamical systems-can be formulated as BPCI problems. Conformal Prediction (CP), a distribution-free technique for uncertainty quantification, has gained significant attention in recent years and has been applied to various control systems problems, particularly to address uncertainties in learned dynamics or controllers. A variant known as training-conditional CP was recently employed to tackle the problem of safety certification. In this note, we highlight that the use of training-conditional CP in this context does not provide valid safety guarantees. We demonstrate why CP is unsuitable for BPCI problems and argue that traditional BPCI methods are better suited for statistical safety certification.