A Spin Glass Characterization of Neural Networks

TL;DR

This paper introduces a novel statistical mechanics approach using spin glass models to analyze individual neural networks, revealing structural properties beyond traditional metrics and aiding in model inspection and safety verification.

Contribution

It presents a new spin glass characterization method for neural networks that provides computable descriptors for individual models, connecting physics concepts with neural network analysis.

Findings

Overlaps between replica samples characterize neural network properties.

The method uncovers structural features not visible through conventional metrics.

Preliminary results indicate potential for practical applications like safety verification.

Abstract

This work presents a statistical mechanics characterization of neural networks, motivated by the replica symmetry breaking (RSB) phenomenon in spin glasses. A Hopfield-type spin glass model is constructed from a given feedforward neural network (FNN). Overlaps between simulated replica samples serve as a characteristic descriptor of the FNN. The connection between the spin-glass description and commonly studied properties of the FNN -- such as data fitting, capacity, generalization, and robustness -- has been investigated and empirically demonstrated. Unlike prior analytical studies that focus on model ensembles, this method provides a computable descriptor for individual network instances, which reveals nontrivial structural properties that are not captured by conventional metrics such as loss or accuracy. Preliminary results suggests its potential for practical applications such as model inspection, safety verification, and detection of hidden vulnerabilities.