Stochastic Approximation with Block Coordinate Optimal Stepsizes

TL;DR

This paper introduces adaptive block-coordinate stepsize rules for stochastic approximation, including a new method that rivals Adam's performance with less memory and fewer hyperparameters, applicable without convexity or smoothness assumptions.

Contribution

It proposes a novel adaptive stepsize method for stochastic approximation with block coordinates, generalizing Adam and providing convergence guarantees under broad conditions.

Findings

New method achieves competitive performance with Adam

Convergence to a neighborhood of the target point is proven

Method requires less memory and fewer hyperparameters

Abstract

We consider stochastic approximation with block-coordinate stepsizes and propose adaptive stepsize rules that aim to minimize the expected distance from the next iterate to an (unknown) target point. These stepsize rules employ online estimates of the second moment of the search direction along each block coordinate. The popular Adam algorithm can be interpreted as a variant with a specific estimator. By leveraging a simple conditional estimator, we derive a new method that obtains competitive performance against Adam but requires less memory and fewer hyper-parameters. We prove that this family of methods converges almost surely to a small neighborhood of the target point, and the radius of the neighborhood depends on the bias and variance of the second-moment estimator. Our analysis relies on a simple aiming condition that assumes neither convexity nor smoothness, thus has broad applicability.