Sparsity-Constraint Optimization via Splicing Iteration

TL;DR

The paper introduces SCOPE, a convergent iterative algorithm for sparsity-constrained optimization that eliminates hyperparameter tuning and demonstrates superior performance in support recovery across various applications.

Contribution

SCOPE is a novel algorithm that replaces gradient steps with splicing guided by objective values, ensuring convergence without hyperparameter tuning.

Findings

SCOPE achieves linear convergence rate.

SCOPE recovers the true support set when sparsity is correctly specified.

Numerical experiments show SCOPE's superior support recovery performance.

Abstract

Sparsity-constrained optimization underlies many problems in signal processing, statistics, and machine learning. State-of-the-art hard-thresholding (HT) algorithms rely on an appropriately selected continuous step-size parameter to ensure convergence. In this paper, we propose a naturally convergent iterative algorithm, SCOPE (Sparsity-Constrained Optimization via sPlicing itEration). The algorithm is capable of optimizing nonlinear differentiable objective functions that are strongly convex and smooth on low-dimensional subspaces. SCOPE replaces the gradient step with a splicing operation guided directly by the objective value, thereby eliminating the need to tune any continuous hyperparameter. Theoretically, it achieves a linear convergence rate and recovers the true support set when the sparsity level is correctly specified. We also establish parallel theoretical results without relying on restricted-isometry-property-type conditions. We apply SCOPE's versatility and power to solve sparse quadratic optimization, learn sparse classifiers, and recover sparse Markov networks for binary variables. With our C++ implementation of SCOPE, numerical experiments on these tasks show that it achieves superior support recovery performance, confirming both its algorithmic efficiency and theoretical guarantees.