Global convergence rate analysis of unconstrained optimization methods based on probabilistic models

TL;DR

This paper establishes global convergence rates for probabilistic model-based optimization methods, showing they match deterministic methods in complexity up to a constant factor, with improved bounds in convex settings.

Contribution

It provides the first comprehensive analysis of convergence rates for probabilistic first- and second-order optimization methods, extending deterministic results to stochastic models.

Findings

Probabilistic models increase evaluation complexity by only a constant factor.

Convergence rates in convex and strongly convex cases are improved and match deterministic bounds.

Probabilistic cubic regularization achieves optimal complexity similar to deterministic methods.

Abstract

We present global convergence rates for a line-search method which is based on random first-order models and directions whose quality is ensured only with certain probability. We show that in terms of the order of the accuracy, the evaluation complexity of such a method is the same as its counterparts that use deterministic accurate models; the use of probabilistic models only increases the complexity by a constant, which depends on the probability of the models being good. We particularize and improve these results in the convex and strongly convex case. We also analyze a probabilistic cubic regularization variant that allows approximate probabilistic second-order models and show improved complexity bounds compared to probabilistic first-order methods; again, as a function of the accuracy, the probabilistic cubic regularization bounds are of the same (optimal) order as for the deterministic case.