Random design analysis of ridge regression

TL;DR

This paper provides a comprehensive analysis of ridge regression and ordinary least squares estimators in random design settings, highlighting prediction errors and effects of covariance and modeling errors.

Contribution

It offers sharp, out-of-sample prediction error bounds and insights into the impact of covariance and modeling errors in random design ridge regression.

Findings

Sharp bounds on out-of-sample prediction error

Effects of covariance estimation errors analyzed

Modeling errors impact quantified

Abstract

This work gives a simultaneous analysis of both the ordinary least squares estimator and the ridge regression estimator in the random design setting under mild assumptions on the covariate/response distributions. In particular, the analysis provides sharp results on the ``out-of-sample'' prediction error, as opposed to the ``in-sample'' (fixed design) error. The analysis also reveals the effect of errors in the estimated covariance structure, as well as the effect of modeling errors, neither of which effects are present in the fixed design setting. The proofs of the main results are based on a simple decomposition lemma combined with concentration inequalities for random vectors and matrices.