Precise analysis of ridge interpolators under heavy correlations -- a Random Duality Theory view

TL;DR

This paper uses Random Duality Theory to precisely analyze the prediction risk of various ridge and linear regression estimators in highly correlated models, revealing detailed dependencies on model parameters and confirming known phenomena like double descent.

Contribution

It introduces a novel application of Random Duality Theory to derive exact closed-form risk characterizations for correlated regression estimators, extending prior spectral method results.

Findings

Risk exhibits double-descent behavior with increasing feature/sample ratio

Explicit risk formulas depend on model dimensions and covariance matrices

Results match spectral method findings in special uncorrelated cases

Abstract

We consider fully row/column-correlated linear regression models and study several classical estimators (including minimum norm interpolators (GLS), ordinary least squares (LS), and ridge regressors). We show that \emph{Random Duality Theory} (RDT) can be utilized to obtain precise closed form characterizations of all estimators related optimizing quantities of interest, including the \emph{prediction risk} (testing or generalization error). On a qualitative level out results recover the risk's well known non-monotonic (so-called double-descent) behavior as the number of features/sample size ratio increases. On a quantitative level, our closed form results show how the risk explicitly depends on all key model parameters, including the problem dimensions and covariance matrices. Moreover, a special case of our results, obtained when intra-sample (or time-series) correlations are not present, precisely match the corresponding ones obtained via spectral methods in [6,16,17,24].