Online Learning, Stability, and Stochastic Gradient Descent

TL;DR

This paper introduces CV_on stability for online learning, demonstrating that stochastic gradient descent (SGD) under typical conditions is CV_on stable and exploring its implications for SGD convergence.

Contribution

It extends the concept of stability to online learning, specifically defining and analyzing CV_on stability for SGD.

Findings

SGD with usual hypotheses is CV_on stable

CV_on stability has implications for SGD convergence

Links stability concepts to online learning performance

Abstract

In batch learning, stability together with existence and uniqueness of the solution corresponds to well-posedness of Empirical Risk Minimization (ERM) methods; recently, it was proved that CV_loo stability is necessary and sufficient for generalization and consistency of ERM. In this note, we introduce CV_on stability, which plays a similar note in online learning. We show that stochastic gradient descent (SDG) with the usual hypotheses is CVon stable and we then discuss the implications of CV_on stability for convergence of SGD.