Sharper lower bounds on the performance of the empirical risk minimization algorithm

TL;DR

This paper establishes sharper lower bounds on the excess risk of the empirical risk minimization algorithm by leveraging advanced probabilistic theorems and geometric assumptions, highlighting fundamental limitations in learning performance.

Contribution

It introduces new lower bounds on ERM performance based on Gaussian process theory and geometric conditions, improving understanding of fundamental learning limits.

Findings

Lower bounds depend on Gaussian process supremum and oscillation parameters.

Bounds are sharper under specific geometric assumptions.

Provides theoretical limits on ERM excess risk performance.

Abstract

We present an argument based on the multidimensional and the uniform central limit theorems, proving that, under some geometrical assumptions between the target function $T$ and the learning class $F$, the excess risk of the empirical risk minimization algorithm is lower bounded by \[\frac{\mathbb{E}\sup_{q\in Q}G_q}{\sqrt{n}}\delta,\] where $(G_q)_{q\in Q}$ is a canonical Gaussian process associated with $Q$ (a well chosen subset of $F$) and $\delta$ is a parameter governing the oscillations of the empirical excess risk function over a small ball in $F$.