Some exercises with the Lasso and its compatibility constant

TL;DR

This paper analyzes the Lasso in noiseless settings, computes the compatibility constant for specific designs, and demonstrates its critical role in bounding the prediction error, showing the bounds are tight.

Contribution

It provides explicit calculations of the compatibility constant for special designs and clarifies its influence on prediction error bounds in Lasso analysis.

Findings

Compatibility constant is crucial for prediction error bounds.

Bounds in the literature are tight up to constants.

Noiseless case analysis informs noisy case prediction error.

Abstract

We consider the Lasso for a noiseless experiment where one has observations $X \beta^0$ and uses the penalized version of basis pursuit. We compute for some special designs the compatibility constant, a quantity closely related to the restricted eigenvalue. We moreover show the dependence of the (penalized) prediction error on this compatibility constant. This exercise illustrates that compatibility is necessarily entering into the bounds for the (penalized) prediction error and that the bounds in the literature therefore are - up to constants - tight. We also give conditions that show that in the noisy case the dominating term for the prediction error is given by the prediction error of the noiseless case.