Exploring Loss Landscapes through the Lens of Spin Glass Theory

TL;DR

This paper applies spin glass theory to analyze the loss landscapes of deep neural networks, revealing insights into their structure, symmetry, and generalizability, and introducing protocols to study these complex landscapes.

Contribution

It introduces a novel perspective by modeling DNN loss landscapes using spin glass theory and develops protocols to analyze their structure and generalization properties.

Findings

Loss landscapes exhibit hierarchical structures similar to spin glass states.

Permutation symmetry leads to multiple equivalent minima in the loss landscape.

Flatter minima correlate with better generalization in DNNs.

Abstract

In the past decade, significant strides in deep learning have led to numerous groundbreaking applications. Despite these advancements, the understanding of the high generalizability of deep learning, especially in such an over-parametrized space, remains limited. For instance, in deep neural networks (DNNs), their internal representations, decision-making mechanism, absence of overfitting in an over-parametrized space, superior generalizability, etc., remain less understood. Successful applications are often considered as empirical rather than scientific achievement. This paper delves into the loss landscape of DNNs through the lens of spin glass in statistical physics, a system characterized by a complex energy landscape with numerous metastable states, as a novel perspective in understanding how DNNs work. We investigated the loss landscape of single hidden layer neural networks activated by Rectified Linear Unit (ReLU) function, and introduced several protocols to examine the analogy between DNNs and spin glass. Specifically, we used (1) random walk in the parameter space of DNNs to unravel the structures in their loss landscape; (2) a permutation-interpolation protocol to study the connection between copies of identical regions in the loss landscape due to the permutation symmetry in the hidden layers; (3) hierarchical clustering to reveal the hierarchy among trained solutions of DNNs, reminiscent of the so-called Replica Symmetry Breaking (RSB) phenomenon (i.e. the Parisi solution) in spin glass; (4) finally, we examine the relationship between the ruggedness of DNN's loss landscape and its generalizability, showing an improvement of flattened minima.