Complexity Analysis of the Lasso Regularization Path

TL;DR

This paper investigates the complexity of computing the Lasso regularization path, proving exponential worst-case complexity but also providing an efficient approximate method with guarantees on optimality.

Contribution

It offers a theoretical analysis of the Lasso path complexity and introduces a practical algorithm for approximate paths with bounded error.

Findings

Worst-case complexity is exponential in the number of variables.

An approximate path with O(1/√ε) segments can be computed efficiently.

The approximate path guarantees an ε-duality gap for all points.

Abstract

The regularization path of the Lasso can be shown to be piecewise linear, making it possible to "follow" and explicitly compute the entire path. We analyze in this paper this popular strategy, and prove that its worst case complexity is exponential in the number of variables. We then oppose this pessimistic result to an (optimistic) approximate analysis: We show that an approximate path with at most O(1/sqrt(epsilon)) linear segments can always be obtained, where every point on the path is guaranteed to be optimal up to a relative epsilon-duality gap. We complete our theoretical analysis with a practical algorithm to compute these approximate paths.