Lasso, fractional norm and structured sparse estimation using a Hadamard product parametrization

TL;DR

This paper introduces a simple, stable, and efficient alternating ridge regression algorithm for sparse estimation using a multiplicative reparametrization that expresses certain $L_q$ penalties as sums of $L_2$ penalties, enabling structured sparsity and broad applicability.

Contribution

It proposes a novel reparametrization-based algorithm for $L_q$ penalized regression that improves stability and speed, and extends to structured sparsity and high-dimensional models.

Findings

The algorithm avoids numerical instability issues common in $L_q$ optimization.

It is competitive in speed compared to EM algorithms.

The method can be extended to generalized linear models and structured sparsity scenarios.

Abstract

Using a multiplicative reparametrization, I show that a subclass of $L_q$ penalties with $q\leq 1$ can be expressed as sums of $L_2$ penalties. It follows that the lasso and other norm-penalized regression estimates may be obtained using a very simple and intuitive alternating ridge regression algorithm. As compared to a similarly intuitive EM algorithm for $L_q$ optimization, the proposed algorithm avoids some numerical instability issues and is also competitive in terms of speed. Furthermore, the proposed algorithm can be extended to accommodate sparse high-dimensional scenarios, generalized linear models, and can be used to create structured sparsity via penalties derived from covariance models for the parameters. Such model-based penalties may be useful for sparse estimation of spatially or temporally structured parameters.