Efficient parameter-free restarted accelerated gradient methods for convex and strongly convex optimization

TL;DR

This paper introduces a new parameter-free restarted accelerated gradient method, RPF-SFISTA, and an aggressive regularization approach, A-REG, both designed for convex and strongly convex optimization, demonstrating significant speed improvements over existing methods.

Contribution

The paper presents RPF-SFISTA, a parameter-free restarted accelerated gradient method that adaptively determines restart points, and A-REG, an aggressive regularization technique for convex optimization, both advancing optimization efficiency.

Findings

RPF-SFISTA is 3 to 15 times faster than existing methods.

RPF-SFISTA requires no knowledge of strong convexity or Lipschitz constants.

A-REG effectively solves regularized subproblems with an aggressive approach.

Abstract

This paper develops a new parameter-free restarted method, namely RPF-SFISTA, and a new parameter-free aggressive regularization method, namely A-REG, for solving strongly convex and convex composite optimization problems, respectively. RPF-SFISTA has the major advantage that it requires no knowledge of both the strong convexity parameter of the entire composite objective and the Lipschitz constant of the gradient. Unlike several other restarted first-order methods which restart an accelerated composite gradient (ACG) method after a predetermined number of ACG iterations have been performed, RPF-SFISTA checks a key inequality at each of iterations to determine when to restart. Extensive computational experiments show that RPF-SFISTA is roughly 3 to 15 times faster than other state-of-the-art restarted methods on four important classes of problems. The A-REG method, developed for convex composite optimization, solves each of its strongly convex regularized subproblems according to a stationarity criterion by using the RPF-SFISTA method with a possibly aggressive choice of initial strong convexity estimate. This scheme is thus more aggressive than several other regularization methods which solve their subproblems by running a standard ACG method for a predetermined number of iterations.