Fast Stochastic Second-Order Adagrad for Nonconvex Bound-Constrained Optimization

TL;DR

This paper introduces ADAGB2, a stochastic second-order optimization algorithm for bound-constrained problems, demonstrating its efficiency and convergence guarantees under certain conditions, and analyzing its relation to classical optimality measures.

Contribution

The paper presents ADAGB2, a novel stochastic second-order method for bound-constrained optimization with proven iteration complexity and analysis of optimality measures.

Findings

ADAGB2 achieves $ ilde{O}(rac{1}{\e^2})$ iteration complexity for approximate critical points.

Convergence depends on the gradient oracle's mean square error being sufficiently small.

Unbiased gradient oracles alone may not guarantee convergence within the desired iteration bound.

Abstract

ADAGB2, a generalization of the Adagrad algorithm for stochastic optimization is introduced, which is also applicable to bound-constrained problems and capable of using second-order information when available. It is shown that, given $\delta\in(0,1)$ and $\epsilon\in(0,1]$, the ADAGB2 algorithm needs at most $\calO(\epsilon^{-2})$ iterations to ensure an $\epsilon$-approximate first-order critical point of the bound-constrained problem with probability at least $1-\delta$, provided the average root mean square error of the gradient oracle is sufficiently small. Should this condition fail, it is also shown that the optimality level of iterates is bounded above by this average. The relation between the approximate and true classical projected-gradient-based optimality measures for bound constrained problems is also investigated, and it is shown that merely assuming unbiased gradient oracles may be insufficient to ensure convergence in $\calO(\epsilon^{-2})$ iterations.