Proximal Iteration for Nonlinear Adaptive Lasso

TL;DR

This paper introduces a joint optimization method using proximal gradient techniques to learn penalty coefficients in adaptive Lasso, reducing bias and enabling complex sparsity structures in nonlinear models.

Contribution

It develops a novel proximal gradient approach that treats penalty coefficients as decision variables, enhancing adaptive Lasso's flexibility and effectiveness in complex models.

Findings

The method is competitive in speed and accuracy on synthetic and real datasets.

It effectively reduces bias in parameter estimates.

Demonstrates applicability to nonlinear models in real-world case studies.

Abstract

Augmenting a smooth cost function with an $\ell_1$ penalty allows analysts to efficiently conduct estimation and variable selection simultaneously in sophisticated models and can be efficiently implemented using proximal gradient methods. However, one drawback of the $\ell_1$ penalty is bias: nonzero parameters are underestimated in magnitude, motivating techniques such as the Adaptive Lasso which endow each parameter with its own penalty coefficient. But it's not clear how these parameter-specific penalties should be set in complex models. In this article, we study the approach of treating the penalty coefficients as additional decision variables to be learned in a \textit{Maximum a Posteriori} manner, developing a proximal gradient approach to joint optimization of these together with the parameters of any differentiable cost function. Beyond reducing bias in estimates, this procedure can also encourage arbitrary sparsity structure via a prior on the penalty coefficients. We compare our method to implementations of specific sparsity structures for non-Gaussian regression on synthetic and real datasets, finding our more general method to be competitive in terms of both speed and accuracy. We then consider nonlinear models for two case studies: COVID-19 vaccination behavior and international refugee movement, highlighting the applicability of this approach to complex problems and intricate sparsity structures.