Convergence in quadratic mean of averaged stochastic gradient algorithms without strong convexity nor bounded gradient

TL;DR

This paper establishes explicit quadratic mean error bounds for averaged stochastic gradient algorithms under very weak assumptions, removing the need for strong convexity or bounded gradients.

Contribution

It provides the first explicit convergence bounds for stochastic gradient algorithms without requiring strong convexity or bounded gradients.

Findings

Explicit quadratic mean error bounds are derived.

Convergence is achieved under very weak assumptions.

Results apply to high-dimensional, sequential data settings.

Abstract

Online averaged stochastic gradient algorithms are more and more studied since (i) they can deal quickly with large sample taking values in high dimensional spaces, (ii) they enable to treat data sequentially, (iii) they are known to be asymptotically efficient. In this paper, we focus on giving explicit bounds of the quadratic mean error of the estimates, and this, with very weak assumptions, i.e without supposing that the function we would like to minimize is strongly convex or admits a bounded gradient.