Adam-SHANG: A Convergent Adam-Type Method for Stochastic Smooth Convex Optimization

TL;DR

We introduce Adam-SHANG, a new Adam-type optimization algorithm with convergence guarantees for stochastic smooth convex problems, featuring a curvature-aware correction and practical stepsize rules, validated by experiments.

Contribution

The paper presents Adam-SHANG, a novel Adam-like method with convergence guarantees and a new stepsize rule, applicable to convex and non-convex deep learning tasks.

Findings

Proves convergence in expectation under a spectral bound stepsize.

Introduces a trace-ratio stepsize for practical implementation.

Demonstrates competitive performance against Adam and AdamW.

Abstract

We propose Adam-SHANG, a Lyapunov-guided Adam-type method that couples momentum, adaptive preconditioning, and a curvature-aware correction through a more stable lagged-preconditioner update. For stochastic smooth convex optimization, we prove convergence in expectation under an admissible stepsize condition that can always be satisfied by a conservative spectral bound, without imposing global monotonicity on the second-moment sequence. To obtain a less conservative practical rule, we introduce a computable trace-ratio stepsize, motivated by a local coordinatewise alignment condition. The same structural update is also tested beyond the convex setting with simplified parameters. Experiments validate the predicted stochastic decay and show competitive training performance against Adam and AdamW on deep learning tasks.