Chaining Bounds for Empirical Risk Minimization

TL;DR

This paper advances the theoretical understanding of empirical risk minimization by extending chaining bounds to unbounded noise settings, applicable to various loss functions and providing tight sample complexity results.

Contribution

It introduces a generalized chaining technique for excess risk bounds in unbounded noise scenarios, applicable to multiple loss functions and detailed analysis for linear regression.

Findings

Bounds apply to many loss functions beyond squared loss

Sample complexity bounds are tight up to logarithmic factors

Method handles unbounded noise and estimates without kurtosis assumptions

Abstract

This paper extends the standard chaining technique to prove excess risk upper bounds for empirical risk minimization with random design settings even if the magnitude of the noise and the estimates is unbounded. The bound applies to many loss functions besides the squared loss, and scales only with the sub-Gaussian or subexponential parameters without further statistical assumptions such as the bounded kurtosis condition over the hypothesis class. A detailed analysis is provided for slope constrained and penalized linear least squares regression with a sub-Gaussian setting, which often proves tight sample complexity bounds up to logartihmic factors.