Deep Learning the Ising Model Near Criticality

TL;DR

This study compares shallow and deep neural networks in modeling the 2D Ising system near criticality, finding shallow networks more efficient for representing physical probability distributions in this context.

Contribution

It demonstrates that shallow neural networks outperform deep ones in representing Ising model data near phase transitions, challenging assumptions about the advantages of depth.

Findings

Shallow networks are more accurate near criticality.

Model accuracy depends mainly on the first hidden layer size.

Deep networks do not show significant advantage in this setting.

Abstract

It is well established that neural networks with deep architectures perform better than shallow networks for many tasks in machine learning. In statistical physics, while there has been recent interest in representing physical data with generative modelling, the focus has been on shallow neural networks. A natural question to ask is whether deep neural networks hold any advantage over shallow networks in representing such data. We investigate this question by using unsupervised, generative graphical models to learn the probability distribution of a two-dimensional Ising system. Deep Boltzmann machines, deep belief networks, and deep restricted Boltzmann networks are trained on thermal spin configurations from this system, and compared to the shallow architecture of the restricted Boltzmann machine. We benchmark the models, focussing on the accuracy of generating energetic observables near the phase transition, where these quantities are most difficult to approximate. Interestingly, after training the generative networks, we observe that the accuracy essentially depends only on the number of neurons in the first hidden layer of the network, and not on other model details such as network depth or model type. This is evidence that shallow networks are more efficient than deep networks at representing physical probability distributions associated with Ising systems near criticality.