Sequential convergence of AdaGrad algorithm for smooth convex optimization

TL;DR

This paper proves the convergence of AdaGrad algorithms, both scalar and coordinatewise variants, for smooth convex functions with Lipschitz gradients by establishing a variable metric quasi-Fejér monotonicity property.

Contribution

It introduces a novel convergence proof for AdaGrad algorithms using a variable metric quasi-Fejér monotonicity approach, applicable to smooth convex optimization.

Findings

AdaGrad sequences are convergent for convex functions with Lipschitz gradients.

The proof relies on the quasi-Fejér monotonicity property.

Both scalar and coordinatewise AdaGrad variants are covered.

Abstract

We prove that the iterates produced by, either the scalar step size variant, or the coordinatewise variant of AdaGrad algorithm, are convergent sequences when applied to convex objective functions with Lipschitz gradient. The key insight is to remark that such AdaGrad sequences satisfy a variable metric quasi-Fej\'er monotonicity property, which allows to prove convergence.