From Density Matrices to Phase Transitions in Deep Learning: Spectral Early Warnings and Interpretability

TL;DR

This paper introduces the 2-datapoint reduced density matrix (2RDM) as a novel, efficient observable for detecting phase transitions in deep learning models during training, inspired by quantum chemistry methods.

Contribution

The paper proposes the 2RDM as a new tool for early warning and interpretability of phase transitions in neural networks, with validation across diverse settings.

Findings

Spectral heat capacity signals second-order phase transitions early.

Participation ratio reveals the dimensionality of reorganization.

Top eigenvectors of 2RDM are directly interpretable.

Abstract

A key problem in the modern study of AI is predicting and understanding emergent capabilities in models during training. Inspired by methods for studying reactions in quantum chemistry, we present the ``2-datapoint reduced density matrix". We show that this object provides a computationally efficient, unified observable of phase transitions during training. By tracking the eigenvalue statistics of the 2RDM over a sliding window, we derive two complementary signals: the spectral heat capacity, which we prove provides early warning of second-order phase transitions via critical slowing down, and the participation ratio, which reveals the dimensionality of the underlying reorganization. Remarkably, the top eigenvectors of the 2RDM are directly interpretable making it straightforward to study the nature of the transitions. We validate across four distinct settings: deep linear networks, induction head formation, grokking, and emergent misalignment. We then discuss directions for future work using the 2RDM.