Small Sample Inference for Generalization Error in Classification Using the CUD Bound

TL;DR

This paper introduces a new method for constructing confidence sets for the generalization error in classification tasks with small samples, using a smooth upper bound and bootstrap, outperforming traditional resampling methods.

Contribution

It proposes a novel confidence set construction based on a smooth upper bound and bootstrap, addressing the non-normality issue in small sample generalization error estimation.

Findings

Outperforms traditional resampling methods in small sample scenarios

Provides a computationally efficient algorithm for parametric additive models

Demonstrates superior performance on test and simulated datasets

Abstract

Confidence measures for the generalization error are crucial when small training samples are used to construct classifiers. A common approach is to estimate the generalization error by resampling and then assume the resampled estimator follows a known distribution to form a confidence set [Kohavi 1995, Martin 1996,Yang 2006]. Alternatively, one might bootstrap the resampled estimator of the generalization error to form a confidence set. Unfortunately, these methods do not reliably provide sets of the desired confidence. The poor performance appears to be due to the lack of smoothness of the generalization error as a function of the learned classifier. This results in a non-normal distribution of the estimated generalization error. We construct a confidence set for the generalization error by use of a smooth upper bound on the deviation between the resampled estimate and generalization error. The confidence set is formed by bootstrapping this upper bound. In cases in which the approximation class for the classifier can be represented as a parametric additive model, we provide a computationally efficient algorithm. This method exhibits superior performance across a series of test and simulated data sets.