Elastic Gradient Descent, an Iterative Optimization Method Approximating the Solution Paths of the Elastic Net

TL;DR

This paper introduces elastic gradient descent, an iterative optimization method that approximates elastic net solution paths more efficiently, offering better accuracy and speed compared to existing methods.

Contribution

The paper proposes elastic gradient descent, a novel generalization of gradient descent and forward stagewise regression, with theoretical analysis and empirical validation demonstrating its advantages.

Findings

Elastic gradient descent closely approximates elastic net solution paths.

It is several orders of magnitude faster than traditional methods.

It achieves lower prediction and estimation errors compared to forward stagewise regression.

Abstract

The elastic net combines lasso and ridge regression to fuse the sparsity property of lasso with the grouping property of ridge regression. The connections between ridge regression and gradient descent and between lasso and forward stagewise regression have previously been shown. Similar to how the elastic net generalizes lasso and ridge regression, we introduce elastic gradient descent, a generalization of gradient descent and forward stagewise regression. We theoretically analyze elastic gradient descent and compare it to the elastic net and forward stagewise regression. Parts of the analysis are based on elastic gradient flow, a piecewise analytical construction, obtained for elastic gradient descent with infinitesimal step size. We also compare elastic gradient descent to the elastic net on real and simulated data and show that it provides similar solution paths, but is several orders of magnitude faster. Compared to forward stagewise regression, elastic gradient descent selects a model that, although still sparse, provides considerably lower prediction and estimation errors.