High-dimensional manifold of solutions in neural networks: insights from statistical physics

TL;DR

This paper reviews the statistical physics approach to neural networks, especially perceptrons, analyzing the geometry of solution spaces, phase transitions, and implications for algorithmic hardness and solution connectivity.

Contribution

It synthesizes recent advances in understanding the geometric structure of neural network solutions using statistical physics methods, highlighting the role of solution space organization.

Findings

Zero training error configurations are geometrically arranged and change with training set size.

Algorithmic hardness is linked to the disappearance of large, clustered solution regions.

Linear mode connectivity provides insights into the shape of the solution manifold.

Abstract

In these pedagogic notes I review the statistical mechanics approach to neural networks, focusing on the paradigmatic example of the perceptron architecture with binary an continuous weights, in the classification setting. I will review the Gardner's approach based on replica method and the derivation of the SAT/UNSAT transition in the storage setting. Then, I discuss some recent works that unveiled how the zero training error configurations are geometrically arranged, and how this arrangement changes as the size of the training set increases. I also illustrate how different regions of solution space can be explored analytically and how the landscape in the vicinity of a solution can be characterized. I give evidence how, in binary weight models, algorithmic hardness is a consequence of the disappearance of a clustered region of solutions that extends to very large distances. Finally, I demonstrate how the study of linear mode connectivity between solutions can give insights into the average shape of the solution manifold.