Stability and Generalization of Stochastic Optimization with Nonconvex and Nonsmooth Problems

TL;DR

This paper provides a systematic stability and generalization analysis for stochastic optimization in nonconvex and nonsmooth problems, introducing novel measures and bounds that connect stability to generalization errors.

Contribution

It introduces new stability measures and establishes their connection to generalization gaps in nonconvex, nonsmooth stochastic optimization, extending analysis to Moreau envelopes and adaptive algorithms.

Findings

Quantitative bounds on stability measures for stochastic algorithms.

Error bounds for stochastic gradient descent and adaptive variants.

Insights into implicit regularization via step size and iteration tuning.

Abstract

Stochastic optimization has found wide applications in minimizing objective functions in machine learning, which motivates a lot of theoretical studies to understand its practical success. Most of existing studies focus on the convergence of optimization errors, while the generalization analysis of stochastic optimization is much lagging behind. This is especially the case for nonconvex and nonsmooth problems often encountered in practice. In this paper, we initialize a systematic stability and generalization analysis of stochastic optimization on nonconvex and nonsmooth problems. We introduce novel algorithmic stability measures and establish their quantitative connection on the gap between population gradients and empirical gradients, which is then further extended to study the gap between the Moreau envelope of the empirical risk and that of the population risk. To our knowledge, these quantitative connection between stability and generalization in terms of either gradients or Moreau envelopes have not been studied in the literature. We introduce a class of sampling-determined algorithms, for which we develop bounds for three stability measures. Finally, we apply these discussions to derive error bounds for stochastic gradient descent and its adaptive variant, where we show how to achieve an implicit regularization by tuning the step sizes and the number of iterations.