Learning without Concentration

TL;DR

This paper derives sharp bounds for Empirical Risk Minimization in convex classes with squared loss, using a small-ball assumption that applies to heavy-tailed functions and targets, improving classical bounds.

Contribution

It introduces a novel analysis method based on small-ball assumptions, removing the need for boundedness or tail decay conditions in ERM performance bounds.

Findings

Bounds scale correctly with noise level

Improves upon classical bounds in bounded scenarios

Applicable to heavy-tailed functions and targets

Abstract

We obtain sharp bounds on the performance of Empirical Risk Minimization performed in a convex class and with respect to the squared loss, without assuming that class members and the target are bounded functions or have rapidly decaying tails. Rather than resorting to a concentration-based argument, the method used here relies on a `small-ball' assumption and thus holds for classes consisting of heavy-tailed functions and for heavy-tailed targets. The resulting estimates scale correctly with the `noise level' of the problem, and when applied to the classical, bounded scenario, always improve the known bounds.