A new non-convex framework to improve asymptotical knowledge on generic stochastic gradient descent

TL;DR

This paper introduces a novel theoretical framework based on Kurdyka-Lojasiewicz theory to analyze the almost-sure convergence of stochastic gradient descent methods in non-convex optimization, providing new asymptotic guarantees.

Contribution

The paper presents a new Kurdyka-Lojasiewicz framework that establishes almost-sure convergence results for SGD in non-convex settings under mild conditions.

Findings

New convergence guarantees for SGD in non-convex optimization

Illustrations through toy simulation examples

Analysis of the impact of theoretical assumptions on SGD behavior

Abstract

Stochastic gradient optimization methods are broadly used to minimize non-convex smooth objective functions, for instance when training deep neural networks. However, theoretical guarantees on the asymptotic behaviour of these methods remain scarce. Especially, ensuring almost-sure convergence of the iterates to a stationary point is quite challenging. In this work, we introduce a new Kurdyka-Lojasiewicz theoretical framework to analyze asymptotic behavior of stochastic gradient descent (SGD) schemes when minimizing non-convex smooth objectives. In particular, our framework provides new almost-sure convergence results, on iterates generated by any SGD method satisfying mild conditional descent conditions. We illustrate the proposed framework by means of several toy simulation examples. We illustrate the role of the considered theoretical assumptions, and investigate how SGD iterates are impacted whether these assumptions are either fully or partially satisfied.