Accelerated projected gradient algorithms for sparsity constrained optimization problems

TL;DR

This paper introduces two accelerated projected gradient algorithms for sparsity-constrained optimization, achieving significant speedups in solving nonconvex best subset selection problems with convergence guarantees.

Contribution

The paper proposes novel acceleration schemes for projected gradient methods that leverage problem structure and subspace identification, improving efficiency and convergence in sparsity-constrained optimization.

Findings

Algorithms are significantly faster than non-accelerated methods.

Proposed methods achieve superlinear convergence in identified subspaces.

Experiments confirm superior performance over state-of-the-art approaches.

Abstract

We consider the projected gradient algorithm for the nonconvex best subset selection problem that minimizes a given empirical loss function under an $\ell_0$-norm constraint. Through decomposing the feasible set of the given sparsity constraint as a finite union of linear subspaces, we present two acceleration schemes with global convergence guarantees, one by same-space extrapolation and the other by subspace identification. The former fully utilizes the problem structure to greatly accelerate the optimization speed with only negligible additional cost. The latter leads to a two-stage meta-algorithm that first uses classical projected gradient iterations to identify the correct subspace containing an optimal solution, and then switches to a highly-efficient smooth optimization method in the identified subspace to attain superlinear convergence. Experiments demonstrate that the proposed accelerated algorithms are magnitudes faster than their non-accelerated counterparts as well as the state of the art.