Projected Gradient Methods with Momentum

TL;DR

This paper analyzes projected gradient methods with momentum for smooth, possibly nonconvex constrained optimization, proposing a new algorithm with theoretical guarantees and improved empirical performance.

Contribution

It introduces a novel momentum-augmented projected gradient algorithm with proven convergence and demonstrates its effectiveness over standard methods.

Findings

The new method outperforms standard projected gradient in benchmarks.

Theoretical analysis confirms convergence guarantees.

Momentum integration improves optimization efficiency in constrained problems.

Abstract

We focus on the optimization problem with smooth, possibly nonconvex objectives and a convex constraint set for which the Euclidean projection operation is practically available. Focusing on this setting, we carry out a general convergence and complexity analysis for algorithmic frameworks. Consequently, we discuss theoretically sound strategies to integrate momentum information within classical projected gradient type algorithms. One of these approaches is then developed in detail, up to the definition of a tailored algorithm with both theoretical guarantees and reasonable per-iteration cost. The proposed method is finally shown to outperform the standard (spectral) projected gradient method in two different experimental benchmarks, indicating that the addition of momentum terms is as beneficial in the constrained setting as it is in the unconstrained scenario.