Do we really need the Rademacher complexities?

TL;DR

This paper challenges the traditional reliance on Rademacher complexities for sample complexity estimates in convex learning with squared loss, showing that Gaussian process behavior is the key factor, leading to a universality result.

Contribution

It demonstrates that sample complexity depends on the limiting Gaussian process rather than Rademacher complexities, providing a new universal perspective for convex learning problems.

Findings

Sample complexity governed by Gaussian processes.

Universality across heavy-tailed distributions.

First such universality result for convex learning.

Abstract

We study the fundamental problem of learning with respect to the squared loss in a convex class. The state-of-the-art sample complexity estimates in this setting rely on Rademacher complexities, which are generally difficult to control. We prove that, contrary to prevailing belief and under minimal assumptions, the sample complexity is not governed by the Rademacher complexities but rather by the behaviour of the limiting gaussian process. In particular, all such learning problems that have the same $L_2$-structure -- even those with heavy-tailed distributions -- share the same sample complexity. This constitutes the first universality result for general convex learning problems. The proof is based on a novel learning procedure, and its performance is studied by combining optimal mean estimation techniques for real-valued random variables with Talagrand's generic chaining method.