Concentration behavior of the penalized least squares estimator

TL;DR

This paper investigates the concentration properties of penalized least squares estimators in nonparametric regression, providing exponential inequalities and bounds that elucidate the estimator's behavior relative to the true function.

Contribution

It introduces exponential concentration inequalities for penalized least squares estimators and derives bounds for key quantities, enhancing understanding of their statistical properties.

Findings

Exponential concentration inequality for the estimator's deviation

Bounds for the nonrandom quantity related to the estimator

Application to smoothing splines example

Abstract

Consider the standard nonparametric regression model and take as estimator the penalized least squares function. In this article, we study the trade-off between closeness to the true function and complexity penalization of the estimator, where complexity is described by a seminorm on a class of functions. First, we present an exponential concentration inequality revealing the concentration behavior of the trade-off of the penalized least squares estimator around a nonrandom quantity, where such quantity depends on the problem under consideration. Then, under some conditions and for the proper choice of the tuning parameter, we obtain bounds for this nonrandom quantity. We illustrate our results with some examples that include the smoothing splines estimator.