Efficient globalization of heavy-ball type methods for unconstrained optimization based on curve searches

TL;DR

This paper introduces a novel globalization strategy for heavy-ball methods in unconstrained optimization, using curve searches to ensure convergence and optimal complexity, with promising practical and theoretical results.

Contribution

It proposes a curve search-based globalization framework for heavy-ball methods, providing global convergence guarantees and optimal complexity bounds in nonconvex optimization.

Findings

The method guarantees global convergence even with nonmonotone decrease conditions.

It achieves optimal worst-case complexity bounds for nonconvex problems.

Preliminary experiments show practical advantages over classical safeguard approaches.

Abstract

In this work, we deal with unconstrained nonlinear optimization problems. Specifically, we are interested in methods carrying out updates possibly along directions not of descent, like Polyak's heavy-ball algorithm. Instead of enforcing convergence properties through line searches and modifications of search direction when suitable safeguards are not satisfied, we propose a strategy based on searches along curve paths: a curve search starting from the first tentative update allows to smoothly revert towards a gradient-related direction if a sufficient decrease condition is not met. The resulting algorithm provably possesses global convergence guarantees, even with a nonmonotone decrease condition. While the presented framework is rather general, particularly of interest is the case of parabolic searches; in this case, under reasonable assumptions, the resulting algorithm can be shown to possess optimal worst case complexity bounds for reaching approximate stationarity in nonconvex settings. Practically, we show that the proposed globalization strategy allows to consistently accept (optimal) pure heavy-ball steps in the strongly convex case, while standard globalization approaches would at times negate them before even evaluating the objective function. Preliminary computational experiments also suggest that the proposed framework might be more convenient than classical safeguard based approaches.