On the boundedness of the sequence generated by minibatch stochastic gradient descent

TL;DR

This paper investigates the conditions under which the sequence generated by minibatch stochastic gradient descent remains bounded, extending previous results to broader classes of functions including coercive functions.

Contribution

It generalizes the boundedness results of SGD with Polyak's stepsize to include coercive functions, beyond the previously known strong convexity case.

Findings

Boundedness holds for a broader class of functions including coercive functions.

A case is presented where boundedness may or may not hold.

Extends theoretical understanding of SGD convergence properties.

Abstract

Stochastic Gradient Descent (SGD) with Polyak's stepsize has recently gained renewed attention in stochastic optimization. Recently, Orvieto, Lacoste-Julien, and Loizou introduced a decreasing variant of Polyak's stepsize, where convergence relies on a boundedness assumption of the iterates. They established that this assumption holds under strong convexity. In this paper, we extend their result by proving that boundedness also holds for a broader class of objective functions, including coercive functions. We also present a case in which boundedness may or may not hold.