Generalization Properties and Implicit Regularization for Multiple Passes SGM

TL;DR

This paper investigates how multiple passes of stochastic gradient methods influence generalization in convex learning, highlighting the roles of step-size and data passes as implicit regularizers, supported by theoretical and numerical evidence.

Contribution

It demonstrates that step-size and number of passes serve as implicit regularization controls in stochastic gradient methods for convex learning.

Findings

Step-size and passes over data affect stability and approximation.

Implicit regularization can be achieved without penalizations or constraints.

Numerical results support theoretical insights.

Abstract

We study the generalization properties of stochastic gradient methods for learning with convex loss functions and linearly parameterized functions. We show that, in the absence of penalizations or constraints, the stability and approximation properties of the algorithm can be controlled by tuning either the step-size or the number of passes over the data. In this view, these parameters can be seen to control a form of implicit regularization. Numerical results complement the theoretical findings.