Alternating Iteratively Reweighted $\ell_1$ and Subspace Newton Algorithms for Nonconvex Sparse Optimization

TL;DR

This paper introduces a hybrid algorithm combining reweighted $ ext{l}_1$ minimization and subspace Newton steps for efficient nonconvex sparse optimization, with proven convergence and superior empirical performance.

Contribution

The paper proposes a novel hybrid method that alternates between reweighted $ ext{l}_1$-regularized subproblems and Newton steps, improving convergence and efficiency in nonconvex sparse optimization.

Findings

Proven global convergence to critical points.

Achieves local linear and quadratic convergence rates.

Outperforms existing methods in efficiency and solution quality.

Abstract

This paper presents a novel hybrid algorithm for minimizing the sum of a continuously differentiable loss function and a nonsmooth, possibly nonconvex, sparse regularization function. The proposed method alternates between solving a reweighted $\ell_1$-regularized subproblem and performing an inexact subspace Newton step. The reweighted $\ell_1$-subproblem allows for efficient closed-form solutions via the soft-thresholding operator, avoiding the computational overhead of proximity operator calculations. As the algorithm approaches an optimal solution, it maintains a stable support set, ensuring that nonzero components stay uniformly bounded away from zero. It then switches to a perturbed regularized Newton method, further accelerating the convergence. We prove global convergence to a critical point and, under suitable conditions, demonstrate that the algorithm exhibits local linear and quadratic convergence rates. Numerical experiments show that our algorithm outperforms existing methods in both efficiency and solution quality across various model prediction problems.