Accelerate the Warm-up Stage in the Lasso Computation via a Homotopic Approach

TL;DR

This paper introduces a homotopic approach with surrogate functions to accelerate the convergence of Lasso estimators, achieving a faster theoretical rate and improved empirical performance over existing algorithms.

Contribution

The paper proposes a novel homotopic method with carefully defined surrogate functions that guarantees a convergence rate of O((log(1/ε))^2), surpassing existing methods.

Findings

Achieves a convergence rate of O((log(1/ε))^2) for Lasso computation.

Demonstrates superior empirical performance compared to state-of-the-art algorithms.

Provides theoretical proof of faster numerical convergence for the proposed method.

Abstract

In optimization, it is known that when the objective functions are strictly convex and well-conditioned, gradient-based approaches can be extremely effective, e.g., achieving the exponential rate of convergence. On the other hand, the existing Lasso-type estimator in general cannot achieve the optimal rate due to the undesirable behavior of the absolute function at the origin. A homotopic method is to use a sequence of surrogate functions to approximate the $\ell_1$ penalty that is used in the Lasso-type of estimators. The surrogate functions will converge to the $\ell_1$ penalty in the Lasso estimator. At the same time, each surrogate function is strictly convex, which enables a provable faster numerical rate of convergence. In this paper, we demonstrate that by meticulously defining the surrogate functions, one can prove a faster numerical convergence rate than any existing methods in computing for the Lasso-type of estimators. Namely, the state-of-the-art algorithms can only guarantee $O(1/\epsilon)$ or $O(1/\sqrt{\epsilon})$ convergence rates, while we can prove an $O([\log(1/\epsilon)]^2)$ for the newly proposed algorithm. Our numerical simulations show that the new algorithm also performs better empirically.