Nonconvex Sparse Learning via Stochastic Optimization with Progressive Variance Reduction

TL;DR

This paper introduces a stochastic variance reduction algorithm for sparse learning with cardinality constraints, achieving fast convergence and high accuracy, and extends it to parallel computing with demonstrated efficiency.

Contribution

The paper presents a novel stochastic variance reduced optimization method for sparse learning, with theoretical convergence guarantees and an asynchronous parallel extension.

Findings

Linear convergence guarantees under certain conditions

Optimal estimation accuracy in high-dimensional settings

Efficient parallel implementation with near linear speedup

Abstract

We propose a stochastic variance reduced optimization algorithm for solving sparse learning problems with cardinality constraints. Sufficient conditions are provided, under which the proposed algorithm enjoys strong linear convergence guarantees and optimal estimation accuracy in high dimensions. We further extend the proposed algorithm to an asynchronous parallel variant with a near linear speedup. Numerical experiments demonstrate the efficiency of our algorithm in terms of both parameter estimation and computational performance.