Optimistic Rates for Learning with a Smooth Loss

TL;DR

This paper derives new excess risk bounds for empirical risk minimization using smooth loss functions, showing improved learning rates that depend on the smoothness and complexity of the hypothesis class.

Contribution

It introduces novel excess risk bounds for smooth loss functions in ERM, online, and stochastic convex optimization, with explicit rates depending on smoothness and complexity.

Findings

Achieves an O(H R_n^2 + R_n  H L*) excess risk bound.

For typical classes, obtains an O(RH/n) rate in the separable case.

Provides guarantees for online and stochastic convex optimization with smooth objectives.

Abstract

We establish an excess risk bound of O(H R_n^2 + R_n \sqrt{H L*}) for empirical risk minimization with an H-smooth loss function and a hypothesis class with Rademacher complexity R_n, where L* is the best risk achievable by the hypothesis class. For typical hypothesis classes where R_n = \sqrt{R/n}, this translates to a learning rate of O(RH/n) in the separable (L*=0) case and O(RH/n + \sqrt{L^* RH/n}) more generally. We also provide similar guarantees for online and stochastic convex optimization with a smooth non-negative objective.