Asymptotic Optimism of Random-Design Linear and Kernel Regression Models

TL;DR

This paper derives the asymptotic optimism for linear and kernel regression models, demonstrating its use as a complexity measure and revealing differences in neural network behavior compared to kernel models.

Contribution

It provides a theoretical foundation for using scaled optimism as a model complexity measure and empirically compares neural networks with kernel models.

Findings

Neural networks with ReLUs behave differently from kernel models under scaled optimism.

Theoretical derivation of asymptotic optimism for linear and kernel regression models.

Empirical methods to compute optimism for real data regression models.

Abstract

We derived the closed-form asymptotic optimism of linear regression models under random designs, and generalizes it to kernel ridge regression. Using scaled asymptotic optimism as a generic predictive model complexity measure, we studied the fundamental different behaviors of linear regression model, tangent kernel (NTK) regression model and three-layer fully connected neural networks (NN). Our contribution is two-fold: we provided theoretical ground for using scaled optimism as a model predictive complexity measure; and we show empirically that NN with ReLUs behaves differently from kernel models under this measure. With resampling techniques, we can also compute the optimism for regression models with real data.