Screening for Sparse Online Learning

TL;DR

This paper introduces a screening rule for online learning algorithms that enables finite identification of sparse structures, leading to computational acceleration in high-dimensional supervised learning tasks.

Contribution

It proposes a novel screening method that, when combined with online algorithms, enforces finite activity identification and improves efficiency.

Findings

Significant acceleration in online sparse learning algorithms.

Effective elimination of irrelevant features during training.

Enhanced exploitation of sparsity for computational gains.

Abstract

Sparsity promoting regularizers are widely used to impose low-complexity structure (e.g. l1-norm for sparsity) to the regression coefficients of supervised learning. In the realm of deterministic optimization, the sequence generated by iterative algorithms (such as proximal gradient descent) exhibit "finite activity identification", namely, they can identify the low-complexity structure in a finite number of iterations. However, most online algorithms (such as proximal stochastic gradient descent) do not have the property owing to the vanishing step-size and non-vanishing variance. In this paper, by combining with a screening rule, we show how to eliminate useless features of the iterates generated by online algorithms, and thereby enforce finite activity identification. One consequence is that when combined with any convergent online algorithm, sparsity properties imposed by the regularizer can be exploited for computational gains. Numerically, significant acceleration can be obtained.