Proximal Quasi-Newton Methods for Regularized Convex Optimization with Linear and Accelerated Sublinear Convergence Rates

TL;DR

This paper analyzes the convergence of proximal quasi-Newton methods for strongly convex composite optimization, introduces practical stopping criteria, and compares accelerated and regular variants, finding acceleration often offers no benefit.

Contribution

The paper provides a convergence analysis for both exact and inexact proximal quasi-Newton methods in strongly convex settings and evaluates an accelerated variant with relaxed assumptions.

Findings

Accelerated proximal quasi-Newton methods may not outperform regular methods.

A practical stopping criterion for subproblem optimization is proposed.

Theoretical analysis confirms sublinear and linear convergence rates.

Abstract

In [19], a general, inexact, efficient proximal quasi-Newton algorithm for composite optimization problems has been proposed and a sublinear global convergence rate has been established. In this paper, we analyze the convergence properties of this method, both in the exact and inexact setting, in the case when the objective function is strongly convex. We also investigate a practical variant of this method by establishing a simple stopping criterion for the subproblem optimization. Furthermore, we consider an accelerated variant, based on FISTA [1], to the proximal quasi-Newton algorithm. A similar accelerated method has been considered in [7], where the convergence rate analysis relies on very strong impractical assumptions. We present a modified analysis while relaxing these assumptions and perform a practical comparison of the accelerated proximal quasi- Newton algorithm and the regular one. Our analysis and computational results show that acceleration may not bring any benefit in the quasi-Newton setting.