A Tight Lower Bound for Uniformly Stable Algorithms

TL;DR

This paper establishes a tight lower bound on the generalization error for uniformly stable algorithms, matching existing upper bounds and advancing theoretical understanding of algorithmic stability in learning theory.

Contribution

It provides the first nontrivial lower bound for uniform stability, confirming the optimality of known upper bounds up to logarithmic factors.

Findings

Proves a tight lower bound of order Ω(γ + L/√n) for uniformly stable algorithms.

Shows the lower bound matches the best known upper bounds up to logarithmic factors.

Fills a significant gap in the theoretical understanding of stability-based generalization bounds.

Abstract

Leveraging algorithmic stability to derive sharp generalization bounds is a classic and powerful approach in learning theory. Since Vapnik and Chervonenkis [1974] first formalized the idea for analyzing SVMs, it has been utilized to study many fundamental learning algorithms (e.g., $k$-nearest neighbors [Rogers and Wagner, 1978], stochastic gradient method [Hardt et al., 2016], linear regression [Maurer, 2017], etc). In a recent line of great works by Feldman and Vondrak [2018, 2019] as well as Bousquet et al. [2020b], they prove a high probability generalization upper bound of order $\tilde{\mathcal{O}}(\gamma +\frac{L}{\sqrt{n}})$ for any uniformly $\gamma$-stable algorithm and $L$-bounded loss function. Although much progress was achieved in proving generalization upper bounds for stable algorithms, our knowledge of lower bounds is rather limited. In fact, there is no nontrivial lower bound known ever since the study of uniform stability [Bousquet and Elisseeff, 2002], to the best of our knowledge. In this paper we fill the gap by proving a tight generalization lower bound of order $\Omega(\gamma+\frac{L}{\sqrt{n}})$, which matches the best known upper bound up to logarithmic factors