Acceleration for Polyak-{\L}ojasiewicz Functions with a Gradient Aiming Condition

TL;DR

This paper investigates the acceleration capabilities of momentum algorithms on Polyak-Łojasiewicz functions, revealing that additional structure like strong quasar-convexity does not guarantee improved bounds, but an aiming condition can enable acceleration.

Contribution

The paper introduces an aiming condition for PL functions that clarifies when momentum algorithms can be provably accelerated, challenging previous assumptions about structural sufficiency.

Findings

Strong quasar-convexity does not ensure acceleration benefits.

An aiming condition on descent directions enables acceleration for PL functions.

Momentum can be accelerated under specific geometric conditions.

Abstract

It is known that when minimizing smooth Polyak-{\L}ojasiewicz (PL) functions, momentum algorithms cannot significantly improve the convergence bound of gradient descent, contrasting with the acceleration phenomenon occurring in the strongly convex case. To bridge this gap, the literature has proposed strongly quasar-convex functions as an intermediate non-convex class, for which accelerated bounds have been suggested to persist. We show that this is not true in general: the additional structure of strong quasar-convexity does not suffice to guaranty better worst-case bounds for momentum compared to gradient descent. As an alternative, we study PL functions under an aiming condition that measures how well the descent direction points toward a minimizer. This perspective clarifies the geometric ingredient enabling provable acceleration by momentum when minimizing PL functions.