A regularized Newton method for $\ell_q$-norm composite optimization problems

TL;DR

This paper introduces HpgSRN, a hybrid Newton method for nonconvex, nonsmooth $ ext{l}_q$-norm regularized problems, proving convergence to stationary points with linear and superlinear rates, and demonstrating superior empirical performance.

Contribution

It proposes a novel hybrid algorithm combining proximal gradient and subspace Newton methods with convergence guarantees for $ ext{l}_q$-norm regularized problems.

Findings

HpgSRN converges to an $L$-stationary point under mild conditions.

The method achieves linear convergence if a Kurdyka-{ extL}ojasiewicz property with exponent 1/2 holds.

Numerical results show HpgSRN is faster and yields better sparsity and objective values than existing methods.

Abstract

This paper is concerned with $\ell_q\,(0<q<1)$-norm regularized minimization problems with a twice continuously differentiable loss function. For this class of nonconvex and nonsmooth composite problems, many algorithms have been proposed to solve them and most of which are of the first-order type. In this work, we propose a hybrid of proximal gradient method and subspace regularized Newton method, named HpgSRN. The whole iterate sequence produced by HpgSRN is proved to have a finite length and converge to an $L$-type stationary point under a mild curve-ratio condition and the Kurdyka-{\L}ojasiewicz property of the cost function, which does linearly if further a Kurdyka-{\L}ojasiewicz property of exponent $1/2$ holds. Moreover, a superlinear convergence rate for the iterate sequence is also achieved under an additional local error bound condition. Our convergence results do not require the isolatedness and strict local minimality properties of the $L$-stationary point. Numerical comparisons with ZeroFPR, a hybrid of proximal gradient method and quasi-Newton method for the forward-backward envelope of the cost function, proposed in [A. Themelis, L. Stella, and P. Patrinos, {\em SIAM J. Optim., } 28(2018), pp. 2274-2303] for the $\ell_q$-norm regularized linear and logistic regressions on real data indicate that HpgSRN not only requires much less computing time but also yields comparable even better sparsities and objective function values.