Empirical and computer-aided robustness analysis of long-step and accelerated methods in smooth convex optimization

TL;DR

This paper evaluates the robustness of various first-order convex optimization methods under gradient inexactness, combining theoretical analysis with empirical testing, and introduces a semi-heuristic shortening factor to enhance method robustness.

Contribution

It provides the first comprehensive robustness analysis of long-step and accelerated methods in inexact settings, proposing a new shortening factor to improve their theoretical guarantees.

Findings

Accelerated methods are more robust than previously expected.

The semi-heuristic shortening factor improves long-step methods' robustness.

Shortened methods show promising performance in inexact gradient scenarios.

Abstract

This work assesses both empirically and theoretically, using the performance estimation methodology, how robust different first-order optimization methods are when subject to relative inexactness in their gradient computations. Relative inexactness occurs, for example, when compressing the gradient using fewer bits of information, which happens when dealing with large-scale problems on GPUs. Three major families of methods are analyzed: constant step gradient descent, long-step methods, and accelerated methods. The latter two are first shown to be theoretically not robust to inexactness. Then, a semi-heuristic shortening factor is introduced to improve their theoretical guarantees. All methods are subsequently tested on a concrete inexact problem, with two different types of relative inexactness, and it is observed that both accelerated methods are much more robust than expected, and that the shortening factor significantly helps the long-step methods. In the end, all shortened methods appear to be promising, even in this inexact setting.