Free Decompression with Algebraic Spectral Curves

TL;DR

This paper introduces an algebraic spectral curve-based method for free decompression, enabling spectral extrapolation across matrix sizes in realistic machine learning models.

Contribution

It generalizes free decompression to handle complex spectral densities with multiple features, improving applicability to real-world ML models.

Findings

Effective spectral density expansion for neural network Hessians.

Successful extrapolation on large-scale diffusion models.

Handles multi-modal and multi-scale spectral data.

Abstract

Tools from random matrix theory have become central to deep learning theory, using spectral information to provide mechanisms for modeling generalization, robustness, scaling, and failure modes. While often capable of modeling empirical behavior, practical computations are limited by matrix size, often imposing a restriction to models that are too small to be realistic. This motivates the inference of properties of larger models from the behavior of smaller ones. Free decompression (FD) is a recently proposed method for extrapolating spectral information across matrix sizes, but its utility is currently limited by strong assumptions that preclude its implementation on more realistic machine learning (ML) models. We use algebraic spectral curve theory to provide a general FD methodology for spectral densities whose Stieltjes transform satisfies an algebraic relation, a modeling assumption that is more likely to hold in practice. This recasts FD as an evolution along spectral curves which can be readily integrated. Our framework enables the expansion of spectral densities that have multiple or multi-modal bulks, that exist at multiple scales, and that contain atoms, all characteristic of real-world data and popular ML models. We demonstrate the efficacy of our framework on models of interest in modern ML, including Hessian and activation matrices associated with neural networks and large-scale diffusion models.