Local Linear Convergence of Forward-Backward under Partial Smoothness

TL;DR

This paper analyzes the local linear convergence of the Forward-Backward algorithm for convex optimization problems with partly smooth functions, providing a unified theoretical framework that explains observed numerical behaviors.

Contribution

It introduces a generic framework showing finite identification of active manifolds and characterizes the local linear convergence of the Forward-Backward algorithm under partial smoothness.

Findings

Finite identification of the active manifold in a finite number of iterations.

Linear convergence rate after manifold identification.

Applicability to problems like Lasso, group Lasso, fused Lasso, and nuclear norm regularization.

Abstract

In this paper, we consider the Forward--Backward proximal splitting algorithm to minimize the sum of two proper convex functions, one of which having a Lipschitz continuous gradient and the other being partly smooth relative to an active manifold $\mathcal{M}$. We propose a generic framework under which we show that the Forward--Backward (i) correctly identifies the active manifold $\mathcal{M}$ in a finite number of iterations, and then (ii) enters a local linear convergence regime that we characterize precisely. This gives a grounded and unified explanation to the typical behaviour that has been observed numerically for many problems encompassed in our framework, including the Lasso, the group Lasso, the fused Lasso and the nuclear norm regularization to name a few. These results may have numerous applications including in signal/image processing processing, sparse recovery and machine learning.