Phase Transitions of Neural Networks

TL;DR

This paper explores phase transitions in neural networks, showing how interactions and competition between error and entropy can cause abrupt changes in network behavior across various models.

Contribution

It applies statistical physics models to analyze neural network phenomena, revealing phase transition behaviors in diverse neural network architectures and learning scenarios.

Findings

Discontinuous phase transitions observed in perceptrons and associative memory.

Critical points identified in learning and generalization processes.

Insights into noise estimation and time series generation through phase transition analysis.

Abstract

The cooperative behaviour of interacting neurons and synapses is studied using models and methods from statistical physics. The competition between training error and entropy may lead to discontinuous properties of the neural network. This is demonstrated for a few examples: Perceptron, associative memory, learning from examples, generalization, multilayer networks, structure recognition, Bayesian estimate, on-line training, noise estimation and time series generation.