Nonsmooth optimization using Taylor-like models: error bounds, convergence, and termination criteria

TL;DR

This paper develops a theoretical framework for nonsmooth optimization algorithms based on Taylor-like models, establishing relationships between step-sizes, convergence, and error bounds, with practical termination criteria and convergence guarantees.

Contribution

It provides explicit links between step-sizes and function slopes, enabling reliable termination and convergence analysis for nonsmooth optimization methods.

Findings

Step-sizes can reliably determine algorithm termination.

Limit points of the iterates are stationary if step-sizes tend to zero.

Error bounds imply linear convergence rates.

Abstract

We consider optimization algorithms that successively minimize simple Taylor-like models of the objective function. Methods of Gauss-Newton type for minimizing the composition of a convex function and a smooth map are common examples. Our main result is an explicit relationship between the step-size of any such algorithm and the slope of the function at a nearby point. Consequently, we (1) show that the step-sizes can be reliably used to terminate the algorithm, (2) prove that as long as the step-sizes tend to zero, every limit point of the iterates is stationary, and (3) show that conditions, akin to classical quadratic growth, imply that the step-sizes linearly bound the distance of the iterates to the solution set. The latter so-called error bound property is typically used to establish linear (or faster) convergence guarantees. Analogous results hold when the step-size is replaced by the square root of the decrease in the model's value. We complete the paper with extensions to when the models are minimized only inexactly.