Newton acceleration on manifolds identified by proximal-gradient methods

TL;DR

This paper demonstrates how combining Riemannian Newton-like methods with proximal gradient steps, which identify differentiability manifolds, can significantly accelerate convergence in nonsmooth optimization problems.

Contribution

It introduces a novel approach that leverages manifold identification by proximal methods to enhance convergence speed using Riemannian Newton-like algorithms.

Findings

Superlinear convergence achieved in certain nonsmooth nonconvex problems

Numerical results show improved convergence on $ ext{l}_1$-norm and trace-norm regularized problems

Manifold identification enables effective acceleration of proximal methods

Abstract

Proximal methods are known to identify the underlying substructure of nonsmooth optimization problems. Even more, in many interesting situations, the output of a proximity operator comes with its structure at no additional cost, and convergence is improved once it matches the structure of a minimizer. However, it is impossible in general to know whether the current structure is final or not; such highly valuable information has to be exploited adaptively. To do so, we place ourselves in the case where a proximal gradient method can identify manifolds of differentiability of the nonsmooth objective. Leveraging this manifold identification, we show that Riemannian Newton-like methods can be intertwined with the proximal gradient steps to drastically boost the convergence. We prove the superlinear convergence of the algorithm when solving some nondegenerated nonsmooth nonconvex optimization problems. We provide numerical illustrations on optimization problems regularized by $\ell_1$-norm or trace-norm.