Generalisation and benign over-fitting for linear regression onto random functional covariates

TL;DR

This paper analyzes the predictive performance of ridge regression with functional covariates evaluated at random locations, revealing how convergence rates depend on model parameters and noise, especially in benign overfitting scenarios.

Contribution

It provides the first theoretical bounds on predictive risk for linear regression with random functional covariates evaluated at unobserved locations, extending understanding beyond i.i.d. data assumptions.

Findings

Convergence rates depend on the growth of p relative to n.

Additive noise influences benign overfitting.

Probabilistic bounds are established under regularity conditions.

Abstract

We study theoretical predictive performance of ridge and ridge-less least-squares regression when covariate vectors arise from evaluating $p$ random, means-square continuous functions over a latent metric space at $n$ random and unobserved locations, subject to additive noise. This leads us away from the standard assumption of i.i.d. data to a setting in which the $n$ covariate vectors are exchangeable but not independent in general. Under an assumption of independence across dimensions, $4$-th order moment, and other regularity conditions, we obtain probabilistic bounds on a notion of predictive excess risk adapted to our random functional covariate setting, making use of recent results of Barzilai and Shamir. We derive convergence rates in regimes where $p$ grows suitably fast relative to $n$, illustrating interplay between ingredients of the model in determining convergence behaviour and the role of additive covariate noise in benign-overfitting.