Average Curvature FISTA for Nonconvex Smooth Composite Optimization Problems

TL;DR

This paper introduces AC-FISTA, an improved accelerated gradient method for nonconvex smooth composite optimization that uses average curvature estimates and fewer evaluations, leading to better practical performance.

Contribution

It proposes AC-FISTA, a novel FISTA-type accelerated method that estimates local curvature using previous observed curvatures and reduces computational effort per iteration.

Findings

AC-FISTA outperforms earlier ACG variants in practice.

It uses fewer composite resolvent evaluations per iteration.

The method effectively estimates local curvature for nonconvex problems.

Abstract

A previous authors' paper introduces an accelerated composite gradient (ACG) variant, namely AC-ACG, for solving nonconvex smooth composite optimization (N-SCO) problems. In contrast to other ACG variants, AC-ACG estimates the local upper curvature of the N-SCO problem by using the average of the observed upper-Lipschitz curvatures obtained during the previous iterations, and uses this estimation and two composite resolvent evaluations to compute the next iterate. This paper presents an alternative FISTA-type ACG variant, namely AC-FISTA, which has the following additional features: i) it performs an average of one composite resolvent evaluation per iteration; and ii) it estimates the local upper curvature by using the average of the previously observed upper (instead of upper-Lipschitz) curvatures. These two properties acting together yield a practical AC-FISTA variant which substantially outperforms earlier ACG variants, including the AC-ACG variants discussed in the aforementioned authors' paper.