Conformal Prediction Assessment: A Framework for Conditional Coverage Evaluation and Selection

TL;DR

This paper introduces Conformal Prediction Assessment (CPA), a new framework that evaluates and improves the conditional coverage of conformal predictors by training a reliability estimator and using a novel index for model selection.

Contribution

The paper proposes CPA, a method to assess and enhance conditional coverage in conformal prediction through a supervised learning approach and a new validity index.

Findings

CPA effectively diagnoses local failure modes in coverage.

CC-Select reliably identifies models with better conditional coverage.

Theoretical guarantees support the convergence and consistency of the proposed methods.

Abstract

Conformal prediction provides rigorous distribution-free finite-sample guarantees for marginal coverage under the assumption of exchangeability, but may exhibit systematic undercoverage or overcoverage for specific subpopulations. Assessing conditional validity is challenging, as standard stratification methods suffer from the curse of dimensionality. We propose Conformal Prediction Assessment (CPA), a framework that reframes the evaluation of conditional coverage as a supervised learning task by training a reliability estimator that predicts instance-level coverage probabilities. Building on this estimator, we introduce the Conditional Validity Index (CVI), which decomposes reliability into safety (undercoverage risk) and efficiency (overcoverage cost). We establish convergence rates for the reliability estimator and prove the consistency of CVI-based model selection. Extensive experiments on synthetic and real-world datasets demonstrate that CPA effectively diagnoses local failure modes and that CC-Select, our CVI-based model selection algorithm, consistently identifies predictors with superior conditional coverage performance.