A Reduced-Space Algorithm for Minimizing $\ell_1$-Regularized Convex Functions

TL;DR

This paper introduces a new reduced-space algorithm for efficiently minimizing convex functions with an -regularizer, leveraging support prediction and adaptive subproblem solving to improve convergence and computational performance.

Contribution

The paper proposes a novel reduced-space method that dynamically predicts support variables and employs flexible subproblem solutions, with proven convergence guarantees.

Findings

Demonstrates efficiency on large prediction problems.

Provides convergence guarantees for the proposed method.

Achieves computational improvements over existing techniques.

Abstract

We present a new method for minimizing the sum of a differentiable convex function and an $\ell_1$-norm regularizer. The main features of the new method include: $(i)$ an evolving set of indices corresponding to variables that are predicted to be nonzero at a solution (i.e., the support); $(ii)$ a reduced-space subproblem defined in terms of the predicted support; $(iii)$ conditions that determine how accurately each subproblem must be solved, which allow for Newton, Newton-CG, and coordinate-descent techniques to be employed; $(iv)$ a computationally practical condition that determines when the predicted support should be updated; and $(v)$ a reduced proximal gradient step that ensures sufficient decrease in the objective function when it is decided that variables should be added to the predicted support. We prove a convergence guarantee for our method and demonstrate its efficiency on a large set of model prediction problems.