Out-of-Distribution Detection Should Use Conformal Prediction (and Vice-versa?)

TL;DR

This paper explores the synergy between Out-of-Distribution detection and Conformal Prediction, proposing new metrics and methods that leverage their interplay to improve reliability and interpretability of anomaly detection.

Contribution

It introduces conformal AUROC and conformal FRP@TPR95 metrics, and demonstrates how combining OOD scores with CP enhances detection performance and guarantees.

Findings

New conformal metrics provide probabilistic conservativeness.

Using OOD scores as non-conformity scores improves CP methods.

Applying these techniques on benchmark datasets shows performance gains.

Abstract

Research on Out-Of-Distribution (OOD) detection focuses mainly on building scores that efficiently distinguish OOD data from In Distribution (ID) data. On the other hand, Conformal Prediction (CP) uses non-conformity scores to construct prediction sets with probabilistic coverage guarantees. In this work, we propose to use CP to better assess the efficiency of OOD scores. Specifically, we emphasize that in standard OOD benchmark settings, evaluation metrics can be overly optimistic due to the finite sample size of the test dataset. Based on the work of (Bates et al., 2022), we define new conformal AUROC and conformal FRP@TPR95 metrics, which are corrections that provide probabilistic conservativeness guarantees on the variability of these metrics. We show the effect of these corrections on two reference OOD and anomaly detection benchmarks, OpenOOD (Yang et al., 2022) and ADBench (Han et al., 2022). We also show that the benefits of using OOD together with CP apply the other way around by using OOD scores as non-conformity scores, which results in improving upon current CP methods. One of the key messages of these contributions is that since OOD is concerned with designing scores and CP with interpreting these scores, the two fields may be inherently intertwined.