Complexity of Adagrad and other first-order methods for nonconvex optimization problems with bounds constraints

TL;DR

This paper extends the Adagrad method to constrained nonconvex optimization problems without computing the objective function, providing complexity guarantees and demonstrating practical benefits through initial experiments.

Contribution

It introduces a new trust-region framework for bound-constrained nonconvex optimization that does not require objective function evaluations and extends Adagrad accordingly.

Findings

Extended Adagrad achieves standard complexity rates.

Objective-free approach benefits noisy bound-constrained problems.

Bounds are essentially sharp with curvature-based stepsize.

Abstract

A parametric class of trust-region algorithms for constrained nonconvex optimization is analyzed, where the objective function is never computed. By defining appropriate first-order stationarity criteria, we are able to extend the Adagrad method to the newly considered problem and retrieve the standard complexity rate of the projected gradient method that uses both the gradient and objective function values. Furthermore, we propose an additional iteration-dependent scaling with slightly inferior theoretical guarantees. In both cases, the bounds are essentially sharp, and curvature information can be used to compute the stepsize. Initial experimental results for noisy bound-constrained instances illustrate the benefits of the objective-free approach.