Two-Point Deterministic Equivalence for Stochastic Gradient Dynamics in Linear Models

TL;DR

This paper introduces a deterministic equivalence for stochastic gradient dynamics in high-dimensional linear models, enabling unified analysis of various models' performance.

Contribution

It provides a new deterministic equivalence for the two-point function of a random matrix resolvent, unifying analysis across multiple high-dimensional linear models.

Findings

Derives a novel deterministic equivalence for the two-point function.

Unifies performance analysis for linear regression, kernel regression, and random feature models.

Includes both known and new asymptotic results.

Abstract

We derive a novel deterministic equivalence for the two-point function of a random matrix resolvent. Using this result, we give a unified derivation of the performance of a wide variety of high-dimensional linear models trained with stochastic gradient descent. This includes high-dimensional linear regression, kernel regression, and linear random feature models. Our results include previously known asymptotics as well as novel ones.