Asymptotics for conformal inference

TL;DR

This paper derives the exact asymptotic distribution of the false coverage proportion in conformal inference, enabling better understanding and control of error rates in large-sample settings across various extensions.

Contribution

It provides the first distribution-free asymptotic distribution for FCP in conformal inference, including extensions to novelty detection and distribution shift scenarios.

Findings

FCP converges to a distribution related to the Kolmogorov distribution.

Asymptotic variance decreases as calibration sample size increases relative to test sample.

Extensions allow accurate quantification of errors under distribution shift.

Abstract

Conformal inference is a versatile tool for building prediction sets in regression or classification. We study the false coverage proportion (FCP) in a simultaneous inference setting with a calibration sample of $n$ points and a test sample of $m$ points. We identify the exact, distribution-free, asymptotic distribution of the FCP when both $n$ and $m$ tend to infinity. This shows in particular that FCP control can be achieved by using the well-known Kolmogorov distribution, and puts forward that the asymptotic variance is decreasing in the ratio $n/m$. We then provide a number of extensions by considering the problems of novelty detection, weighted conformal inference or distribution shift between the calibration sample and the test sample. In particular, our asymptotic results allow to accurately quantify the asymptotic behavior of the errors (a miscovering interval or declaring a false novelty) when weighted conformal inference is used.