Local Linear Convergence of ISTA and FISTA on the LASSO Problem

TL;DR

This paper analyzes the local linear convergence behavior of ISTA and FISTA algorithms on the LASSO problem, revealing that FISTA's acceleration diminishes near the solution, suggesting a switch to ISTA for efficiency.

Contribution

It provides a spectral analysis showing the convergence rates of ISTA and FISTA near the solution, and proposes switching strategies based on convergence behavior.

Findings

Both ISTA and FISTA converge linearly near the solution.

FISTA's acceleration effect diminishes close to the solution.

Switching from FISTA to ISTA can improve convergence efficiency.

Abstract

We establish local linear convergence bounds for the ISTA and FISTA iterations on the model LASSO problem. We show that FISTA can be viewed as an accelerated ISTA process. Using a spectral analysis, we show that, when close enough to the solution, both iterations converge linearly, but FISTA slows down compared to ISTA, making it advantageous to switch to ISTA toward the end of the iteration processs. We illustrate the results with some synthetic numerical examples.