Enhanced Cyclic Coordinate Descent Methods for Elastic Net Penalized Linear Models

TL;DR

This paper introduces an enhanced cyclic coordinate descent framework for elastic net penalized linear models that significantly reduces training time through a Taylor expansion-based approximation and batched computations, outperforming existing methods.

Contribution

The paper proposes a novel ECCD method that improves training efficiency and stability for elastic net models by unrolling recurrences and optimizing computations.

Findings

Achieves 3x faster training on benchmark datasets.

Reduces convergence delay and numerical instability.

Provides an open-source C++ implementation.

Abstract

We present a novel enhanced cyclic coordinate descent (ECCD) framework for solving generalized linear models with elastic net constraints that reduces training time in comparison to existing state-of-the-art methods. We redesign the CD method by performing a Taylor expansion around the current iterate to avoid nonlinear operations arising in the gradient computation. By introducing this approximation, we are able to unroll the vector recurrences occurring in the CD method and reformulate the resulting computations into more efficient batched computations. We show empirically that the recurrence can be unrolled by a tunable integer parameter, $s$, such that $s > 1$ yields performance improvements without affecting convergence, whereas $s = 1$ yields the original CD method. A key advantage of ECCD is that it avoids the convergence delay and numerical instability exhibited by block coordinate descent. Finally, we implement our proposed method in C++ using Eigen to accelerate linear algebra computations. Comparison of our method against existing state-of-the-art solvers shows consistent performance improvements of $3\times$ in average for regularization path variant on diverse benchmark datasets. Our implementation is available at https://github.com/Yixiao-Wang-Stats/ECCD.