Recursive Bound-Constrained AdaGrad with Applications to Multilevel and Domain Decomposition Minimization

TL;DR

This paper introduces two noise-tolerant, bound-constrained optimization algorithms based on AdaGrad, utilizing multilevel and domain decomposition techniques, with proven convergence and demonstrated efficiency on PDE and neural network problems.

Contribution

The paper develops two novel algorithms extending AdaGrad for bound-constrained, noisy optimization, with a unified convergence theory and broad application scope.

Findings

Both algorithms require at most O(ε^{-2}) iterations to find an ε-approximate critical point.

Numerical experiments show high efficiency on PDE and deep learning problems.

Algorithms are robust to noise and inexact gradients in complex settings.

Abstract

Two OFFO (Objective-Function Free Optimization) noise tolerant algorithms are presented that handle bound constraints, inexact gradients and use second-order information when available.The first is a multi-level method exploiting a hierarchical description of the problem and the second is a domain-decomposition method covering the standard addditive Schwarz decompositions. Both are generalizations of the first-order AdaGrad algorithm for unconstrained optimization. Because these algorithms share a common theoretical framework, a single convergence/complexity theory is provided which covers them both. Its main result is that, with high probability, both methods need at most $O(\epsilon^{-2})$ iterations and noisy gradient evaluations to compute an $\epsilon$-approximate first-order critical point of the bound-constrained problem. Extensive numerical experiments are discussed on applications ranging from PDE-based problems to deep neural network training, illustrating their remarkable computational efficiency.