Orthant Based Proximal Stochastic Gradient Method for $\ell_1$-Regularized Optimization

TL;DR

This paper introduces OBProx-SG, a novel stochastic optimization method that enhances sparsity and convergence in l1-regularized problems, outperforming existing methods in both convex and non-convex machine learning tasks.

Contribution

The paper proposes a new orthant-based stochastic gradient method that improves sparsity promotion and convergence guarantees for l1-regularized optimization problems.

Findings

OBProx-SG converges to global optima or stationary points.

It significantly enhances sparsity compared to existing methods.

It achieves higher sparsity in deep neural networks without accuracy loss.

Abstract

Sparsity-inducing regularization problems are ubiquitous in machine learning applications, ranging from feature selection to model compression. In this paper, we present a novel stochastic method -- Orthant Based Proximal Stochastic Gradient Method (OBProx-SG) -- to solve perhaps the most popular instance, i.e., the l1-regularized problem. The OBProx-SG method contains two steps: (i) a proximal stochastic gradient step to predict a support cover of the solution; and (ii) an orthant step to aggressively enhance the sparsity level via orthant face projection. Compared to the state-of-the-art methods, e.g., Prox-SG, RDA and Prox-SVRG, the OBProx-SG not only converges to the global optimal solutions (in convex scenario) or the stationary points (in non-convex scenario), but also promotes the sparsity of the solutions substantially. Particularly, on a large number of convex problems, OBProx-SG outperforms the existing methods comprehensively in the aspect of sparsity exploration and objective values. Moreover, the experiments on non-convex deep neural networks, e.g., MobileNetV1 and ResNet18, further demonstrate its superiority by achieving the solutions of much higher sparsity without sacrificing generalization accuracy.