On the Interplay between Acceleration and Identification for the Proximal Gradient algorithm

TL;DR

This paper investigates how acceleration techniques affect structure identification in the proximal gradient algorithm, revealing negative effects and proposing a method to improve stability without sacrificing convergence speed.

Contribution

It introduces a generic method to control acceleration in the proximal gradient algorithm, enhancing stability in structure identification while maintaining optimal convergence rates.

Findings

The interplay can cause oscillations and loss of structure.

The proposed method improves stability in subspace identification.

Convergence rate remains comparable to accelerated methods.

Abstract

In this paper, we study the interplay between acceleration and structure identification for the proximal gradient algorithm. We report and analyze several cases where this interplay has negative effects on the algorithm behavior (iterates oscillation, loss of structure, etc.). We present a generic method that tames acceleration when structure identification may be at stake; it benefits from a convergence rate that matches the one of the accelerated proximal gradient under some qualifying condition. We show empirically that the proposed method is much more stable in terms of subspace identification compared to the accelerated proximal gradient method while keeping a similar functional decrease.