A globally convergent fast iterative shrinkage-thresholding algorithm with a new momentum factor for single and multi-objective convex optimization

TL;DR

This paper introduces a new accelerated proximal gradient method with a flexible momentum factor for convex optimization, achieving a proven $O(1/k^2)$ convergence rate and demonstrating improved practical performance over classical methods.

Contribution

It proposes a novel momentum parameterization for FISTA that guarantees convergence and enhances finite-time convergence properties in both single and multi-objective convex optimization.

Findings

The method attains a global $O(1/k^2)$ convergence rate for all hyperparameters.

Generated iterates converge to a weak Pareto solution when the hyperparameter $a$ is positive.

Numerical experiments show some hyperparameter choices outperform classical momentum factors.

Abstract

Convex-composite optimization, which minimizes an objective function represented by the sum of a differentiable function and a convex one, is widely used in machine learning and signal/image processing. Fast Iterative Shrinkage Thresholding Algorithm (FISTA) is a typical method for solving this problem and has a global convergence rate of $O(1 / k^2)$. Recently, this has been extended to multi-objective optimization, together with the proof of the $O(1 / k^2)$ global convergence rate. However, its momentum factor is classical, and the convergence of its iterates has not been proven. In this work, introducing some additional hyperparameters $(a, b)$, we propose another accelerated proximal gradient method with a general momentum factor, which is new even for the single-objective cases. We show that our proposed method also has a global convergence rate of $O(1/k^2)$ for any $(a,b)$, and further that the generated sequence of iterates converges to a weak Pareto solution when $a$ is positive, an essential property for the finite-time manifold identification. Moreover, we report numerical results with various $(a,b)$, showing that some of these choices give better results than the classical momentum factors.