Singular ridge regression with homoscedastic residuals: generalization error with estimated parameters

TL;DR

This paper derives explicit formulas for the generalization and regression errors of singular ridge regression estimators, accounting for estimation inaccuracies under a homoscedastic residual assumption, without assuming a solution exists for the non-regularized problem.

Contribution

It provides the first explicit formulas for errors in singular ridge regression under a fully singular setup with homoscedastic residuals, extending classical results.

Findings

Explicit formulas for total and generalization errors.

Error analysis in fully singular ridge regression.

Error bounds under homoscedastic residuals.

Abstract

This paper characterizes the conditional distribution properties of the finite sample ridge regression estimator and uses that result to evaluate total regression and generalization errors that incorporate the inaccuracies committed at the time of parameter estimation. The paper provides explicit formulas for those errors. Unlike other classical references in this setup, our results take place in a fully singular setup that does not assume the existence of a solution for the non-regularized regression problem. In exchange, we invoke a conditional homoscedasticity hypothesis on the regularized regression residuals that is crucial in our developments.