Optimal upper and lower bounds for the true and empirical excess risks in heteroscedastic least-squares regression

TL;DR

This paper derives optimal finite-sample bounds for true and empirical excess risks of least-squares estimators in heteroscedastic regression, establishing their equivalence and conditions for consistency in sup-norm.

Contribution

It provides the first nonasymptotic, optimal bounds for excess risks in heteroscedastic regression and links their equivalence to estimator consistency in sup-norm.

Findings

Bounds are nonasymptotic and optimal to first order.

True and empirical excess risks are equivalent under certain conditions.

Sup-norm consistency is established for histogram and piecewise polynomial models.

Abstract

We consider the estimation of a bounded regression function with nonparametric heteroscedastic noise and random design. We study the true and empirical excess risks of the least-squares estimator on finite-dimensional vector spaces. We give upper and lower bounds on these quantities that are nonasymptotic and optimal to first order, allowing the dimension to depend on sample size. These bounds show the equivalence between the true and empirical excess risks when, among other things, the least-squares estimator is consistent in sup-norm with the projection of the regression function onto the considered model. Consistency in the sup-norm is then proved for suitable histogram models and more general models of piecewise polynomials that are endowed with a localized basis structure.