Hessian Spectral Analysis at Foundation Model Scale

TL;DR

This paper demonstrates that accurate Hessian spectral analysis of large foundation models is feasible at scale, revealing limitations of common approximations and enabling more principled curvature-based analysis.

Contribution

It introduces a scalable method for spectral analysis of the true Hessian at model scales up to 100B parameters, surpassing prior small-model limitations.

Findings

Hessian spectra can be accurately estimated at large scale.

Block-diagonal curvature approximations can be significantly inaccurate.

Spectral analysis incurs modest overhead over first-order training.

Abstract

Accurate Hessian spectra of foundation models have remained out of reach, leading most prior work to rely on small models or strong structural approximations. We show that faithful spectral analysis of the true Hessian is tractable at frontier scale. Using shard-local finite-difference Hessian vector products compatible with Fully Sharded Data Parallelism, we perform stochastic Lanczos quadrature on open-source language models with up to 100B parameters, producing the first large-scale spectral density estimates beyond the sub-10B regime. We characterize the numerical behavior of this pipeline, including finite-difference bias, floating-point noise amplification, and their effect on Krylov stability in fp32 and bf16, and derive practical operating regimes that are validated empirically. We further provide end-to-end runtime and memory scaling laws, showing that full-operator spectral probing incurs only a modest constant-factor overhead over first-order training. Crucially, direct access to the Hessian reveals that widely used block-diagonal curvature approximations can fail catastrophically, exhibiting order-one relative error and poor directional alignment even in mid-scale LLMs. Together, our results demonstrate that foundation-model Hessian spectra are both computable and qualitatively misrepresented by prevailing approximations, opening the door to principled curvature-based analysis at scale.