Conditional Coverage Diagnostics for Conformal Prediction

TL;DR

This paper introduces a new classification-based approach to evaluate and diagnose conditional coverage in conformal prediction, addressing limitations of existing metrics and providing a practical tool for assessing local reliability.

Contribution

It proposes the excess risk of the target coverage (ERT) metric, leveraging modern classifiers to improve detection of coverage deviations and benchmarking conformal methods.

Findings

ERT outperforms traditional metrics in statistical power

Modern classifiers improve detection of coverage violations

Open-source package for ERT and other metrics is released

Abstract

Evaluating conditional coverage remains one of the most persistent challenges in assessing the reliability of predictive systems. Although conformal methods can give guarantees on marginal coverage, no method can guarantee to produce sets with correct conditional coverage, leaving practitioners without a clear way to interpret local deviations. To overcome sample-inefficiency and overfitting issues of existing metrics, we cast conditional coverage estimation as a classification problem. Conditional coverage is violated if and only if any classifier can achieve lower risk than the target coverage. Through the choice of a (proper) loss function, the resulting risk difference gives a conservative estimate of natural miscoverage measures such as L1 and L2 distance, and can even separate the effects of over- and under-coverage, and non-constant target coverages. We call the resulting family of metrics excess risk of the target coverage (ERT). We show experimentally that the use of modern classifiers provides much higher statistical power than simple classifiers underlying established metrics like CovGap. Additionally, we use our metric to benchmark different conformal prediction methods. Finally, we release an open-source package for ERT as well as previous conditional coverage metrics. Together, these contributions provide a new lens for understanding, diagnosing, and improving the conditional reliability of predictive systems.