Resampled Confidence Regions with Exponential Shrinkage for the Regression Function of Binary Classification

TL;DR

This paper develops distribution-free confidence regions for the regression function in binary classification using resampling tests, applicable to broad model classes, with theoretical guarantees and practical illustrations.

Contribution

It introduces a novel empirical risk minimization approach for confidence regions with strong consistency and exponential bounds, applicable to arbitrary model classes.

Findings

Strong uniform consistency for finite pseudo-dimension models

Exponential PAC bounds on confidence region sizes

Pointwise bounds for k-nearest neighbors method

Abstract

The regression function is one of the key objects of binary classification, since it not only determines a Bayes optimal classifier, hence, defines an optimal decision boundary, but also encodes the conditional distribution of the output given the input. In this paper we build distribution-free confidence regions for the regression function for any user-chosen confidence level and any finite sample size based on a resampling test. These regions are abstract, as the model class can be almost arbitrary, e.g., it does not have to be finitely parameterized. We prove the strong uniform consistency of a new empirical risk minimization based approach for model classes with finite pseudo-dimensions and inverse Lipschitz parameterizations. We provide exponential probably approximately correct bounds on the $L_2$ sizes of these regions, and demonstrate the ideas on specific models. Additionally, we also consider a k-nearest neighbors based method, for which we prove strong pointwise bounds on the probability of exclusion. Finally, the constructions are illustrated on a logistic model class and compared to the asymptotic ellipsoids of the maximum likelihood estimator.