Estimating Subagging by cross-validation

TL;DR

This paper derives concentration inequalities for cross-validation estimates of generalization error in subagged estimators, applicable to various procedures and predictor classes, providing bounds that improve understanding of their reliability.

Contribution

It generalizes existing formalism to cover many cross-validation methods and offers new bounds on the probability of estimation errors, along with a simple subbagging rule.

Findings

Probability bounds are tighter, combining Hoeffding and Vapnik bounds.

Bounds are valid even for small training sets.

Provides a practical rule for subbagging predictors.

Abstract

In this article, we derive concentration inequalities for the cross-validation estimate of the generalization error for subagged estimators, both for classification and regressor. General loss functions and class of predictors with both finite and infinite VC-dimension are considered. We slightly generalize the formalism introduced by \cite{DUD03} to cover a large variety of cross-validation procedures including leave-one-out cross-validation, $k$-fold cross-validation, hold-out cross-validation (or split sample), and the leave-$\upsilon$-out cross-validation. \bigskip \noindent An interesting consequence is that the probability upper bound is bounded by the minimum of a Hoeffding-type bound and a Vapnik-type bounds, and thus is smaller than 1 even for small learning set. Finally, we give a simple rule on how to subbag the predictor. \bigskip