Uniform Convergence with Square-Root Lipschitz Loss

TL;DR

This paper provides uniform convergence guarantees for Gaussian data using Rademacher complexity and square-root Lipschitz constants, broadening previous smoothness-based results to include non-smooth losses like those in phase retrieval and ReLU regression.

Contribution

It introduces a generalization of uniform convergence bounds to square-root Lipschitz losses, extending applicability to non-smooth functions and improving understanding of optimistic rate and interpolation learning.

Findings

Guarantees hold for non-smooth losses like ReLU and phase retrieval

Generalizes previous smoothness-based uniform convergence results

Offers insights into optimistic rate and interpolation learning

Abstract

We establish generic uniform convergence guarantees for Gaussian data in terms of the Rademacher complexity of the hypothesis class and the Lipschitz constant of the square root of the scalar loss function. We show how these guarantees substantially generalize previous results based on smoothness (Lipschitz constant of the derivative), and allow us to handle the broader class of square-root-Lipschitz losses, which includes also non-smooth loss functions appropriate for studying phase retrieval and ReLU regression, as well as rederive and better understand "optimistic rate" and interpolation learning guarantees.