One-step estimator paths for concave regularization

TL;DR

This paper introduces POSE, a fast algorithm for sparse diminishing-bias regularization that adapts lasso paths with coefficient-specific weights, improving estimation accuracy without extra computational cost.

Contribution

It proposes a new one-step estimator path framework that enhances lasso regularization with adaptive weights, enabling better sparse diminishing-bias estimation efficiently.

Findings

POSE performs comparably or better than existing methods in simulations.

The gamma lasso implementation provides reliable degrees of freedom estimates.

Application to hockey data demonstrates practical utility.

Abstract

The statistics literature of the past 15 years has established many favorable properties for sparse diminishing-bias regularization: techniques which can roughly be understood as providing estimation under penalty functions spanning the range of concavity between $L_0$ and $L_1$ norms. However, lasso $L_1$-regularized estimation remains the standard tool for industrial `Big Data' applications because of its minimal computational cost and the presence of easy-to-apply rules for penalty selection. In response, this article proposes a simple new algorithm framework that requires no more computation than a lasso path: the path of one-step estimators (POSE) does $L_1$ penalized regression estimation on a grid of decreasing penalties, but adapts coefficient-specific weights to decrease as a function of the coefficient estimated in the previous path step. This provides sparse diminishing-bias regularization at no extra cost over the fastest lasso algorithms. Moreover, our `gamma lasso' implementation of POSE is accompanied by a reliable heuristic for the fit degrees of freedom, so that standard information criteria can be applied in penalty selection. We also provide novel results on the distance between weighted-$L_1$ and $L_0$ penalized predictors; this allows us to build intuition about POSE and other diminishing-bias regularization schemes. The methods and results are illustrated in extensive simulations and in application of logistic regression to evaluating the performance of hockey players.