Improved Pathwise Coordinate Descent for Power Penalties

TL;DR

This paper introduces improved pathwise coordinate descent algorithms for $ ext{l}_q$ penalized regression, enabling faster computation of solution paths for nonconvex penalties and potentially better solutions.

Contribution

The paper presents a reparameterization approach that enhances pathwise coordinate descent algorithms for $ ext{l}_q$ penalties, including nonconvex cases, and introduces two efficient algorithms.

Findings

Pathwise algorithms outperform cold-start methods in speed.

Different algorithms may find better solutions depending on the scenario.

Reparameterization simplifies the computation of the mode-thresholding function.

Abstract

Pathwise coordinate descent algorithms have been used to compute entire solution paths for lasso and other penalized regression problems quickly with great success. They improve upon cold start algorithms by solving the problems that make up the solution path sequentially for an ordered set of tuning parameter values, instead of solving each problem separately. However, extending pathwise coordinate descent algorithms to more the general bridge or power family of $\ell_q$ penalties is challenging. Faster algorithms for computing solution paths for these penalties are needed because $\ell_q$ penalized regression problems can be nonconvex and especially burdensome to solve. In this paper, we show that a reparameterization of $\ell_q$ penalized regression problems is more amenable to pathwise coordinate descent algorithms. This allows us to improve computation of the mode-thresholding function for $\ell_q$ penalized regression problems in practice and introduce two separate pathwise algorithms. We show that either pathwise algorithm is faster than the corresponding cold-start alternative, and demonstrate that different pathwise algorithms may be more likely to reach better solutions.