Stability and Sharper Risk Bounds with Convergence Rate $\tilde{O}(1/n^2)$

TL;DR

This paper improves the theoretical understanding of stability and risk bounds in machine learning, achieving faster convergence rates under common assumptions, and provides the tightest high-probability bounds for nonconvex settings.

Contribution

It establishes sharper risk bounds of order (log^2(n)/n^2) for strongly-convex learners and extends high-probability generalization bounds to nonconvex scenarios.

Findings

Achieves (log^2(n)/n^2) risk bounds under common assumptions.

Provides the tightest high-probability bounds for gradient-based generalization in nonconvex learning.

Improves upon previous (log(n)/n) bounds for strongly-convex learners.

Abstract

Prior work (Klochkov $\&$ Zhivotovskiy, 2021) establishes at most $O\left(\log (n)/n\right)$ excess risk bounds via algorithmic stability for strongly-convex learners with high probability. We show that under the similar common assumptions -- - Polyak-Lojasiewicz condition, smoothness, and Lipschitz continous for losses -- - rates of $O\left(\log^2(n)/n^2\right)$ are at most achievable. To our knowledge, our analysis also provides the tightest high-probability bounds for gradient-based generalization gaps in nonconvex settings.