Bounds on the prediction error of penalized least squares estimators with convex penalty

TL;DR

This paper establishes sharp bounds on the prediction error of convex penalized least squares estimators, demonstrating subgaussian concentration and deriving oracle inequalities that hold at any confidence level.

Contribution

It introduces new concentration results and oracle inequalities for convex penalized estimators like LASSO, with bounds valid at any confidence level and improved concentration rates.

Findings

Prediction error is subgaussian and concentrated around its median.

Oracle inequalities hold at any confidence level for estimators with data-independent tuning.

Concentration rates are faster than previously understood.

Abstract

This paper considers the penalized least squares estimator with arbitrary convex penalty. When the observation noise is Gaussian, we show that the prediction error is a subgaussian random variable concentrated around its median. We apply this concentration property to derive sharp oracle inequalities for the prediction error of the LASSO, the group LASSO and the SLOPE estimators, both in probability and in expectation. In contrast to the previous work on the LASSO type methods, our oracle inequalities in probability are obtained at any confidence level for estimators with tuning parameters that do not depend on the confidence level. This is also the reason why we are able to establish sparsity oracle bounds in expectation for the LASSO type estimators, while the previously known techniques did not allow for the control of the expected risk. In addition, we show that the concentration rate in the oracle inequalities is better than it was commonly understood before.