Concentration inequalities for leave-one-out cross validation

TL;DR

This paper establishes concentration inequalities for leave-one-out cross validation using estimator stability, extending results beyond Lipschitz assumptions and applying to various estimators under distributions satisfying the logarithmic Sobolev inequality.

Contribution

It introduces a general framework for concentration bounds in leave-one-out cross validation that do not require Lipschitz conditions, broadening applicability.

Findings

Provides concentration bounds for leave-one-out CV under estimator stability.

Extends results to distributions satisfying the logarithmic Sobolev inequality.

Applies the framework to linear regression, kernel density estimation, and stabilized estimators.

Abstract

In this article we prove that estimator stability is enough to show that leave-one-out cross validation is a sound procedure, by providing concentration bounds in a general framework. In particular, we provide concentration bounds beyond Lipschitz continuity assumptions on the loss or on the estimator. We obtain our results by relying on random variables with distribution satisfying the logarithmic Sobolev inequality, providing us a relatively rich class of distributions. We illustrate our method by considering several interesting examples, including linear regression, kernel density estimation, and stabilized/truncated estimators such as stabilized kernel regression.