Lecture Notes on Statistical Physics and Neural Networks

TL;DR

This paper provides an accessible introduction to statistical physics concepts relevant for neural networks and deep learning, covering phase transitions, spin models, and modern deep learning architectures.

Contribution

It bridges statistical physics and neural networks, explaining key concepts and models like Boltzmann machines and deep learning motivated by physics principles.

Findings

Introduces the Boltzmann-Gibbs distribution and thermodynamic potentials.

Explains phase transitions as limits of infinite lattice points.

Discusses learning algorithms for restricted Boltzmann machines and their relation to deep learning.

Abstract

These lecture notes introduce some topics of classical statistical physics, particularly those that are relevant for neural networks and deep learning. Statistical physics is treated as a branch of probability theory or statistics, with the goal of making concepts such as phase transitions and the renormalization group accessible to readers without prior knowledge of physics. We introduce the Boltzmann-Gibbs distribution and the thermodynamic potentials on a finite configuration space, notably for Ising spins and spin-glass models on a lattice, and then define phase transitions as discontinuities that arise in the limit that the number of lattice points goes to infinity. We further introduce Hopfield networks and Boltzmann machines, which are governed by the same energy function as spin-glass models, and discuss the learning algorithm for restricted Boltzmann machines. In this algorithm hidden neurons are integrated out as in the renormalization group. Finally, modern deep learning is introduced, whose early developments were in part motivated by restricted Boltzmann machines in that they carry many layers of hidden neurons. A description of large language models is given.