Posterior Conformal Prediction

TL;DR

This paper introduces posterior conformal prediction (PCP), a method that provides both marginal and approximate conditional coverage guarantees for prediction intervals, improving tightness and subgroup-specific validity across various applications.

Contribution

The paper presents PCP, a novel conformal prediction approach that models conditional conformity scores as a mixture of clusters, enabling better subgroup coverage and tighter intervals.

Findings

PCP achieves tighter prediction intervals than existing methods.

PCP provides approximate conditional coverage for discovered subgroups.

In classification, PCP adapts coverage based on classifier confidence.

Abstract

Conformal prediction is a popular technique for constructing prediction intervals with distribution-free coverage guarantees. The coverage is marginal, meaning it only holds on average over the entire population but not necessarily for any specific subgroup. This article introduces a new method, posterior conformal prediction (PCP), which generates prediction intervals with both marginal and approximate conditional validity for clusters (or subgroups) naturally discovered in the data. PCP achieves these guarantees by modelling the conditional conformity score distribution as a mixture of cluster distributions. Compared to other methods with approximate conditional validity, this approach produces tighter intervals, particularly when the test data is drawn from clusters that are well represented in the validation data. PCP can also be applied to guarantee conditional coverage on user-specified subgroups, in which case it achieves robust coverage on smaller subgroups within the specified subgroups. In classification, the theory underlying PCP allows for adjusting the coverage level based on the classifier's confidence, achieving significantly smaller sets than standard conformal prediction sets. We evaluate the performance of PCP on diverse datasets from socio-economic, scientific and healthcare applications.