Asymptotics of Ridge (less) Regression under General Source Condition

TL;DR

This paper provides an asymptotic analysis of ridge regression's prediction error considering the structure of the true parameter, revealing conditions where interpolation can be optimal even at finite SNR.

Contribution

It introduces a novel asymptotic framework for ridge regression under general source conditions, linking parameter structure to optimal regularization strategies.

Findings

Interpolation can be optimal at finite SNR with structured parameters.

The true parameter's alignment with data variance influences regularization choice.

A precise characterization of test error depending on data covariance and parameter structure.

Abstract

We analyze the prediction error of ridge regression in an asymptotic regime where the sample size and dimension go to infinity at a proportional rate. In particular, we consider the role played by the structure of the true regression parameter. We observe that the case of a general deterministic parameter can be reduced to the case of a random parameter from a structured prior. The latter assumption is a natural adaptation of classic smoothness assumptions in nonparametric regression, which are known as source conditions in the the context of regularization theory for inverse problems. Roughly speaking, we assume the large coefficients of the parameter are in correspondence to the principal components. In this setting a precise characterisation of the test error is obtained, depending on the inputs covariance and regression parameter structure. We illustrate this characterisation in a simplified setting to investigate the influence of the true parameter on optimal regularisation for overparameterized models. We show that interpolation (no regularisation) can be optimal even with bounded signal-to-noise ratio (SNR), provided that the parameter coefficients are larger on high-variance directions of the data, corresponding to a more regular function than posited by the regularization term. This contrasts with previous work considering ridge regression with isotropic prior, in which case interpolation is only optimal in the limit of infinite SNR.