Optimization over Sparse Support-Preserving Sets: Two-Step Projection with Global Optimality Guarantees

TL;DR

This paper introduces a new two-step projection method for sparse optimization with additional support-preserving constraints, providing global objective guarantees and improving convergence analysis over existing techniques across various settings.

Contribution

It develops a novel two-step projection algorithm with global optimality guarantees for sparse support-preserving constrained optimization, extending analysis techniques and improving existing zeroth-order results.

Findings

Provides a two-step projection algorithm with global guarantees.

Extends the three-point lemma to analyze non-convex projections.

Improves zeroth-order convergence results by removing non-vanishing errors.

Abstract

In sparse optimization, enforcing hard constraints using the $\ell_0$ pseudo-norm offers advantages like controlled sparsity compared to convex relaxations. However, many real-world applications demand not only sparsity constraints but also some extra constraints. While prior algorithms have been developed to address this complex scenario with mixed combinatorial and convex constraints, they typically require the closed form projection onto the mixed constraints which might not exist, and/or only provide local guarantees of convergence which is different from the global guarantees commonly sought in sparse optimization. To fill this gap, in this paper, we study the problem of sparse optimization with extra support-preserving constraints commonly encountered in the literature. We present a new variant of iterative hard-thresholding algorithm equipped with a two-step consecutive projection operator customized for these mixed constraints, serving as a simple alternative to the Euclidean projection onto the mixed constraint. By introducing a novel trade-off between sparsity relaxation and sub-optimality, we provide global guarantees in objective value for the output of our algorithm, in the deterministic, stochastic, and zeroth-order settings, under the conventional restricted strong-convexity/smoothness assumptions. As a fundamental contribution in proof techniques, we develop a novel extension of the classic three-point lemma to the considered two-step non-convex projection operator, which allows us to analyze the convergence in objective value in an elegant way that has not been possible with existing techniques. In the zeroth-order case, such technique also improves upon the state-of-the-art result from de Vazelhes et. al. (2022), even in the case without additional constraints, by allowing us to remove a non-vanishing system error present in their work.