Proximal Subgradient Norm Minimization of ISTA and FISTA

TL;DR

This paper introduces a new analysis framework for ISTA and FISTA, showing that their proximal subgradient norms converge at inverse square and inverse cubic rates respectively, advancing understanding of acceleration in composite optimization.

Contribution

The paper develops a tighter inequality for step size and Lipschitz constant, leading to a novel Lyapunov-based analysis that proves accelerated convergence rates for ISTA and FISTA.

Findings

ISTA's squared proximal subgradient norm converges at an inverse square rate.

FISTA's squared proximal subgradient norm converges at an inverse cubic rate.

The analysis is applicable regardless of gradient correction or implicit velocity.

Abstract

For first-order smooth optimization, the research on the acceleration phenomenon has a long-time history. Until recently, the mechanism leading to acceleration was not successfully uncovered by the gradient correction term and its equivalent implicit-velocity form. Furthermore, based on the high-resolution differential equation framework with the corresponding emerging techniques, phase-space representation and Lyapunov function, the squared gradient norm of Nesterov's accelerated gradient descent (\texttt{NAG}) method at an inverse cubic rate is discovered. However, this result cannot be directly generalized to composite optimization widely used in practice, e.g., the linear inverse problem with sparse representation. In this paper, we meticulously observe a pivotal inequality used in composite optimization about the step size $s$ and the Lipschitz constant $L$ and find that it can be improved tighter. We apply the tighter inequality discovered in the well-constructed Lyapunov function and then obtain the proximal subgradient norm minimization by the phase-space representation, regardless of gradient-correction or implicit-velocity. Furthermore, we demonstrate that the squared proximal subgradient norm for the class of iterative shrinkage-thresholding algorithms (ISTA) converges at an inverse square rate, and the squared proximal subgradient norm for the class of faster iterative shrinkage-thresholding algorithms (FISTA) is accelerated to convergence at an inverse cubic rate.