Testing for Outliers with Conformal p-values

TL;DR

This paper develops conformal p-value methods for nonparametric outlier detection, providing finite-sample guarantees and a new approach for valid, independent p-values to improve false discovery rate control.

Contribution

It introduces a novel conformal inference framework that produces valid, independent p-values for outlier detection, enabling stronger error control and uniform false positive bounds.

Findings

Proposed conformal p-values are positively dependent and enable FDR control.

Developed a new method for valid, independent p-values conditioned on training data.

Numerical experiments demonstrate effectiveness on real and simulated data.

Abstract

This paper studies the construction of p-values for nonparametric outlier detection, taking a multiple-testing perspective. The goal is to test whether new independent samples belong to the same distribution as a reference data set or are outliers. We propose a solution based on conformal inference, a broadly applicable framework which yields p-values that are marginally valid but mutually dependent for different test points. We prove these p-values are positively dependent and enable exact false discovery rate control, although in a relatively weak marginal sense. We then introduce a new method to compute p-values that are both valid conditionally on the training data and independent of each other for different test points; this paves the way to stronger type-I error guarantees. Our results depart from classical conformal inference as we leverage concentration inequalities rather than combinatorial arguments to establish our finite-sample guarantees. Furthermore, our techniques also yield a uniform confidence bound for the false positive rate of any outlier detection algorithm, as a function of the threshold applied to its raw statistics. Finally, the relevance of our results is demonstrated by numerical experiments on real and simulated data.