Uniformly Valid Confidence Sets Based on the Lasso

TL;DR

This paper develops confidence sets for linear regression parameters based on the Lasso estimator that are valid both in finite samples and asymptotically, addressing estimation uncertainty and post-model selection error.

Contribution

It introduces a method for constructing uniformly valid confidence regions based on the Lasso, with exact formulas for coverage probability and considerations for different tuning regimes.

Findings

Exact formulas for minimal coverage probability in finite samples with Gaussian errors.

Confidence regions that are valid in both finite samples and asymptotic regimes.

Comparison of confidence set shapes and their properties.

Abstract

In a linear regression model of fixed dimension $p \leq n$, we construct confidence regions for the unknown parameter vector based on the Lasso estimator that uniformly and exactly hold the prescribed in finite samples as well as in an asymptotic setup. We thereby quantify estimation uncertainty as well as the "post-model selection error" of this estimator. More concretely, in finite samples with Gaussian errors and asymptotically in the case where the Lasso estimator is tuned to perform conservative model selection, we derive exact formulas for computing the minimal coverage probability over the entire parameter space for a large class of shapes for the confidence sets, thus enabling the construction of valid confidence regions based on the Lasso estimator in these settings. The choice of shape for the confidence sets and comparison with the confidence ellipse based on the least-squares estimator is also discussed. Moreover, in the case where the Lasso estimator is tuned to enable consistent model selection, we give a simple confidence region with minimal coverage probability converging to one. Finally, we also treat the case of unknown error variance and present some ideas for extensions.