Uniform Stability for First-Order Empirical Risk Minimization

TL;DR

This paper develops methods to create uniformly stable first-order optimization algorithms for empirical risk minimization, achieving near-optimal convergence rates and resolving open problems in stability and generalization bounds.

Contribution

It introduces a black-box conversion for Euclidean algorithms to ensure uniform stability and extends Mirror Descent variants to general geometries, advancing stability guarantees.

Findings

Achieves nearly optimal convergence rate of O(1/T^2) with stability O(T^2/n)

Develops a Mirror Descent variant with convergence O(1/T) and stability O(T/n)

Resolves an open problem on stability for smooth optimization algorithms

Abstract

We consider the problem of designing uniformly stable first-order optimization algorithms for empirical risk minimization. Uniform stability is often used to obtain generalization error bounds for optimization algorithms, and we are interested in a general approach to achieve it. For Euclidean geometry, we suggest a black-box conversion which given a smooth optimization algorithm, produces a uniformly stable version of the algorithm while maintaining its convergence rate up to logarithmic factors. Using this reduction we obtain a (nearly) optimal algorithm for smooth optimization with convergence rate $\widetilde{O}(1/T^2)$ and uniform stability $O(T^2/n)$, resolving an open problem of Chen et al. (2018); Attia and Koren (2021). For more general geometries, we develop a variant of Mirror Descent for smooth optimization with convergence rate $\widetilde{O}(1/T)$ and uniform stability $O(T/n)$, leaving open the question of devising a general conversion method as in the Euclidean case.