Local geometry of high-dimensional mixture models: Effective spectral theory and dynamical transitions

TL;DR

This paper analyzes the local geometry of high-dimensional mixture models by deriving spectral properties of Hessian and information matrices, linking training dynamics to spectral transitions in models like logistic regression.

Contribution

It provides exact formulas for spectral limits and eigenstructure in high dimensions, connecting training dynamics to spectral transitions in mixture models.

Findings

Spectral distributions depend on parameter summaries via Gram matrices.

Eigenvalues and eigenvectors exhibit static and dynamical phase transitions.

Effective dynamics describe the evolution of spectral properties during training.

Abstract

We study the local geometry of empirical risks in high dimensions via the spectral theory of their Hessian and information matrices. We focus on settings where the data, $(Y_\ell)_{\ell =1}^n \in \mathbb{R}^d$, are i.i.d. draws of a $k$-Gaussian mixture model, and the loss depends on the projection of the data into a fixed number of vectors, namely $\mathbf{x}^\top Y$, where $\mathbf{x}\in \mathbb{R}^{d\times C}$ are the parameters, and $C$ need not equal $k$. This setting captures a broad class of problems such as classification by one and two-layer networks and regression on multi-index models. We provide exact formulas for the limits of the empirical spectral distribution and outlier eigenvalues and eigenvectors of such matrices in the proportional asymptotics limit, where the number of samples and dimension $n,d\to\infty$ and $n/d=\phi \in (0,\infty)$. These limits depend on the parameters $\mathbf{x}$ only through the summary statistic of the $(C+k)\times (C+k)$ Gram matrix of the parameters and class means, $\mathbf{G} = (\mathbf{x},\boldsymbol{\mu})^\top(\mathbf{x},\boldsymbol{\mu})$. It is known that under general conditions, when $\mathbf{x}$ is trained by online stochastic gradient descent, the evolution of these same summary statistics along training converges to the solution of an autonomous system of ODEs, called the effective dynamics. This enables us to connect the training dynamics to the spectral theory of these matrices generated with test data. We demonstrate our general results by analyzing the effective spectrum along the effective dynamics in the case of multi-class logistic regression. In this setting, the empirical Hessian and information matrices have substantially different spectra, each with their own static and even dynamical spectral transitions.