Rectifying Conformity Scores for Better Conditional Coverage

TL;DR

This paper introduces a trainable transformation for conformity scores in split conformal prediction, enhancing conditional coverage and adaptivity, especially in multi-output problems, with theoretical guarantees and improved empirical performance.

Contribution

It proposes a novel, trainable transformation of conformity scores based on conditional quantile estimates, improving conditional coverage in conformal prediction.

Findings

Outperforms existing methods in conditional coverage

Highly adaptive to local data structures

Provides theoretical bounds on approximate conditional validity

Abstract

We present a new method for generating confidence sets within the split conformal prediction framework. Our method performs a trainable transformation of any given conformity score to improve conditional coverage while ensuring exact marginal coverage. The transformation is based on an estimate of the conditional quantile of conformity scores. The resulting method is particularly beneficial for constructing adaptive confidence sets in multi-output problems where standard conformal quantile regression approaches have limited applicability. We develop a theoretical bound that captures the influence of the accuracy of the quantile estimate on the approximate conditional validity, unlike classical bounds for conformal prediction methods that only offer marginal coverage. We experimentally show that our method is highly adaptive to the local data structure and outperforms existing methods in terms of conditional coverage, improving the reliability of statistical inference in various applications.