Phase transition encoded in neural network

TL;DR

This paper demonstrates that neural networks trained to predict temperature in Ising/Potts models inherently encode phase transition information in their parameters, revealing how physical phenomena are captured by learned features.

Contribution

It shows that neural networks trained for temperature prediction automatically learn and encode phase transition information without explicit supervision.

Findings

Weights in trained networks contain phase transition information.

Networks learn different physical quantities depending on training quality.

Critical temperature can be identified from network parameters.

Abstract

We discuss an aspect of neural networks for the purpose of phase transition detection. To this end, we first train the neural network by feeding Ising/Potts configurations with labels of temperature so that it can predict the temperature of input. We do not explicitly supervise whether the configurations are in ordered/disordered phase. Nevertheless, we can identify the critical temperature from the parameters (weights and biases) of trained neural network. We attempt to understand how temperature-supervised neural networks capture the information of phase transition by paying attention to what quantities they learn. Our detailed analyses reveal that they learn different physical quantities depending on how well they are trained. Main observation in this study is how the weights in the trained neural-network can have information of the phase transition in addition to temperature.

Equations

$$H(\{{\sigma }_i\})=-\sum _{⟨i,j⟩}{\sigma }_i{\sigma }_j,$$

$$\displaystyle\left[\begin{array}[]{l}\mathcal{I}=\Big{\{}\{\sigma_{i}\}\Big{|}\text{ Ising configs on }L\times L\text{ lattice.}\Big{\}}\\ \downarrow\ \left\{\begin{array}[]{ll}\text{Fully-connected (Dense) layer}&\\ \text{Softmax activation}&\\ \end{array}\right.\\ x_{a}\in\text{[0,1]}^{N_{\mathrm{h}}}\text{ : hidden units}\\ \downarrow\ \left\{\begin{array}[]{ll}\text{Fully-connected (Dense) layer}&\\ \text{Softmax activation}&\\ \end{array}\right.\\ y_{K}\in\text{[0,1]}^{N_{\mathrm{o}}}\text{ : output}\end{array}\right]$$

$$x_a=\text{softmax}(w_{ai}^{(1)}{\sigma }_i+b_a^{(1)}):=\frac{{\mathrm{e}}^{w_{ai}^{(1)}{\sigma }_i+b_a^{(1)}}}{{\sum }_a{\mathrm{e}}^{w_{ai}^{(1)}{\sigma }_i+b_a^{(1)}}},$$

$$y_K=\text{softmax}(w_{Ka}^{(2)}{x}_a+b_K^{(2)}).$$

$$K^{\mathrm{output}}\equiv \underset{K}{\mathrm{argmax}}(y_K),$$

$$E(y_K,1_{K=K_i^{\text{target}}})=-\frac{1}{N_{\mathrm{o}}}\sum _i{1}_{K=K_i^{\text{target}}}\ln y_K$$

$$\displaystyle\bm{1}_{K=K^{\text{target}}_{i}}=\left\{\begin{array}[]{ll}1&(K=K_{i}^{\text{target}})\\ 0&(\text{otherwise})\end{array}\right.$$

$$H(\{{\Phi }_i\})=-\sum _{⟨i,j⟩}\delta ({\Phi }_i,{\Phi }_j),$$

$$\tilde{x}=\left(\begin{array}{c}{\tilde{x}}_0\\ {\tilde{x}}_1\\ {\tilde{x}}_2\end{array}\right)=\left(\begin{array}{c}-m\\ ϵ\\ m\end{array}\right),$$

$$m<-ϵ:x=\left(\begin{array}{c}1\\ 0\\ 0\end{array}\right),-ϵ\le m<ϵ:x=\left(\begin{array}{c}0\\ 1\\ 0\end{array}\right),ϵ\le m:x=\left(\begin{array}{c}0\\ 0\\ 1\end{array}\right).$$

$$\text{Low K : }\left(\begin{array}{c}1\\ 0\\ 0\end{array}\right),\text{Critical K : }\left(\begin{array}{c}0\\ 1\\ 0\end{array}\right),\text{High K : }\left(\begin{array}{c}0\\ 0\\ 1\end{array}\right),$$

$$w^{(2)}=\left(\begin{array}{ccc}-\Delta & 0& -\Delta \\ -\delta & -\delta & -\delta \\ 0& -\Delta & 0\end{array}\right),$$

$$m<-ϵ$$

$$-ϵ\le m<ϵ$$

$$ϵ\le m$$

$$\displaystyle\left[\begin{array}[]{l}\mathcal{I}=\Big{\{}\{\sigma_{i}\}\Big{|}\text{ Ising configs on }L\times L\text{ lattice.}\Big{\}}\\ \downarrow\ \left\{\begin{array}[]{ll}\text{Convolution}_{[(s_{1},s_{1})\text{-filter, }(s_{1},s_{1})\text{-stride, }C_{1}\text{-channels}]}&\\ \text{ReLU activation}&\\ \text{Convolution}_{[(s_{2},s_{2})\text{-filter, }(s_{2},s_{2})\text{-stride, }C_{2}\text{-channels}]}&\\ \text{ReLU activation}&\\ \text{Convolution}_{[(s_{3},s_{3})\text{-filter, }(s_{3},s_{3})\text{-stride, }C_{3}\text{-channels}]}&\\ \text{ReLU activation}&\\ \text{Flatten}&\\ \end{array}\right.\\ x_{a}\in\mathbb{R}^{N_{\mathrm{h}}=\ L^{2}/(s_{1}s_{2}s_{3})^{2}\times C_{3}}\text{ : hidden units}\\ \downarrow\ \left\{\begin{array}[]{ll}\text{Fully connected layer}&\\ \text{Softmax activation}&\\ \end{array}\right.\\ y_{\alpha}\in\text{[0,1]}^{N_{\mathrm{t}}}\text{ : output}\end{array}\right]$$

$$x_a=\text{ReLU}({\tilde{x}}_a)=\text{Max}({\tilde{x}}_a,0),$$

$$L=16,(s_1,s_2,s_3)=(2,2,4),(C_1,C_2,C_3)=(64,32,3),$$

$$x_a=\text{ReLU}(w_{ai}^{(1)}{\sigma }_i+b_a^{(1)}),\text{leavevmode}\text{leavevmode}\text{leavevmode}\text{leavevmode}x=\left(\begin{array}{c}{x}_0\\ x_1\\ x_2\end{array}\right)=\left(\begin{array}{c}E+2ϵ\\ 0\\ -\varphi E\end{array}\right),$$

$$P(\{{\sigma }_i\};K)=\frac{{\mathrm{e}}^{-KE(\{{\sigma }_i\})}}{\mathcal{Z}(K)}.$$

$$y_K^{\text{theory}}=\frac{P(\{{\sigma }_i\};K)}{{\sum }_{K^{\prime }}P(\{{\sigma }_i\};K^{\prime })}=\text{softmax}(-KE+F),$$

$$\displaystyle K^{\text{output}}=\operatorname*{arg\,max}_{K}\big{[}\text{softmax}(-KE+F))\big{]}.$$

$$w_{K0}^{(2)}=-\varphi G(K)-pK+q,w_{K2}^{(2)}=-G(K)+rK+s,$$

$$y_K$$

$$\displaystyle w^{(2)}_{K0}=\frac{p+r}{2\epsilon}F(K)+q,\quad w^{(2)}_{K2}=\frac{p+r}{2\epsilon\phi}F(K)+\big{(}\frac{p}{\phi}+r\big{)}K+s.$$

$$⟨E⟩=\frac{{\sum }_{\{{\sigma }_i\}}H(\{{\sigma }_i\}){\mathrm{e}}^{-KH(\{{\sigma }_i\})}}{{\sum }_{\{{\sigma }_i\}}{\mathrm{e}}^{-KH(\{{\sigma }_i\})}}=-\frac{\partial }{\partial K}\ln \mathcal{Z},$$

$$⟨E^2-⟨E{⟩}^2⟩=-\frac{\partial }{\partial K}⟨E⟩=\frac{{\partial }^2}{\partial K^2}\ln \mathcal{Z}.$$

$$f=-\frac{\ln \mathcal{Z}}{N}=-\frac{\ln 2}{2}-\ln [\cosh (2K)]-\frac{1}{2\pi }{\int }_0^{\pi }\mathrm{d}\theta \ln \left[1+\sqrt{1-{\left(\frac{2\sinh (2K)}{{\cosh }^2(2K)}\cos \theta \right)}^2}\right],$$

Full text

I Introduction

Exploring phases of matters is one of the most important tasks to reveal infrared structure of the underlying microscopic physical system. Their phases are classified based on the symmetries that the theory possesses landau1937theory ; ginzburg1950theory . In a theory with several distinct phases, phase transitions occur at their boundaries. Among them, the nature of second order phase transition is solely determined by number of dimensions, global symmetries of underlying theory independent of its microscopic details, i.e., classified by the universality class. In reality, however, analytic determination of various phases or detection of phase transitions based on the data of microscopic theory is generally a hard problem because we mostly find it difficult to exactly solve them or identify the corresponding infrared theory. Therefore, tremendous amount of works have been devoted to unravel them with numerical approaches. An obvious and major obstacle is that the larger the number of degrees of freedom grows the harder the numerical analyses become.

Machine learning has grown up rapidly in the field of computer science and made prominent successes in pattern recognition, image processing, etc. Recently, we have witnessed that the machine learning has also been applied in various branches of physics. Detection of phase transitions is one of the intriguing examples that machine learning may make new progresses, and several approaches have already been proposed and examined in simple models such as spin systems. In those works, the trainings are carried out with supervision carrasquilla2017machine ; Tanaka:2016rtu ; ohtsuki2016deep ; mano2017phase ; schindler2017probing ; broecker2017machine ; ch2017machine ; zhang2017quantum ; ponte2017kernel ; zhang2017machine ; arai2018deep ; beach2018machine ; Wetzel:2017ooo ; richter2018machine ; Suchsland2018 ; Kim2018 or without one wang2016discovering ; morningstar2017deep ; van2017learning ; hu2017discovering ; wetzel2017unsupervised ; Iso:2018yqu . Here, the input data is prepared independently of the neural network or its training process. For instance, the Monte Carlo simulation or experimental data based on the physical system of our interest provide the necessary input data.

One approach to detect a phase boundary of a physical system is a supervised binary classification, where a neural network is trained so that it can distinguish its ordered and disordered phases. Indeed, it reasonably detects the phase transition in several models from their raw data carrasquilla2017machine . More recently, some implicit connections between latent variables and physical quantities have been extensively studied toward the ultimate goal of eliminating black-box nature of machine learning technique carrasquilla2017machine ; Wetzel:2017ooo ; zhang2017machine ; Kim2018 . It also succeeded in detecting non-standard phase transition such as topological phase transitions and BKT phase transition with the help of feature engineerings based on physical insights. zhang2017quantum ; zhang2017machine ; beach2018machine ; richter2018machine ; Zhang2018 ; Sun2018 .

Another intriguing approach was proposed in Tanaka:2016rtu to detect the phase transition by speculating that the information of order parameters are encoded in the weights of neural network as a consequence of training. They attempted to identify the critical temperature of the two dimensional Ising model based on supervised machine learning. A fully-connected as well as convolutional neural network are trained in such a way that it can correctly predict the target temperature of input spin configuration. It is surprising that they succeeded in extracting the phase transition temperature from its weight because they did not feed any direct information about phase transition during the training. It implies that the network spontaneously captures the phase transition and encodes it in the machine parameters along the process of supervised learning of temperature although the underlying mechanism of the outcome was remained unresolved.

The purpose of this article is to understand the mechanism of this phase transition detection. Understanding what this approach captures will be useful when we try to apply it to other unknown systems because it may not detect the phase transition correctly if it is triggered by different physical mechanisms such as quasi long-range order (BKT transition) or topology. Indeed, it turns out that the method captures different physical quantities, i.e., the features of input configurations, depending on how we train the neural network. In order to illustrate the idea, we pay attention to simplified neural network architectures embodying the essence of temperature prediction.

II Critical temperature prediction

We consider two dimensional Ising model, described by the Hamiltonian $H(\{\sigma_{i}\})=-J\sum_{\langle i,j\rangle}\sigma_{i}\sigma_{j}$, where the coupling constant $J$ is taken to be positive. $\sigma_{i}\in\{-1,1\}$ represents a spin degree of freedom living on a site of a square lattice of size $L\times L$. We impose periodic boundary condition on the spin variables. The sum is taken over nearest neighboring sites. We redefine the Hamiltonian as

[TABLE]

by absorbing the coupling constant into the inverse temperature $K:=J/k_{\text{B}}T$ on the Boltzmann weight $e^{-KH}/{\cal Z}$, where $\mathcal{Z}=\sum_{\{\sigma_{i}\}}\mathrm{e}^{-KH(\{\sigma_{i}\})}$ is the partition function.

We attempt to detect its critical temperature associated with the second-order phase transition. Nevertheless, our first step is to construct a thermometer by employing a supervised feed-forward neural network. Then, we will examine the weights and biases of the trained neural networks, and attempt to identify the critical temperatures of two dimensional Ising model and 3-state Potts model.

We generate 2000 Ising spin configurations at each temperature by employing the Metropolis-Hasting algorithm. They are fed to a neural network along with the 20 target temperatures. We implement two types of neural network architectures by using KERAS package with TENSORFLOW as the backend: fully-connected and convolutional neural networks.

The former consists of fully-connected hidden and output layers as follows:

[TABLE]

Let us describe it in detail here. We denote input degrees of freedom by $\{\sigma_{i}\}\,(i=1,\dots,L\times L)$, which would be spins in case of Ising model. A hidden unit $x_{a}\ (a=1,\dots,N_{\mathrm{h}})$ is given by,

[TABLE]

where repeated indices are summed. $w^{(1)}_{ai}$ and $b^{(1)}_{a}$ are weights and biases of the first layer, respectively. In terms of weights $w^{(2)}_{\alpha a}$ and biases $b^{(2)}_{\alpha}$ of the second layer, variables $y_{K}\ (K=1,\dots,N_{\mathrm{o}})$ of output layer takes the same form as the hidden variables,

[TABLE]

Based on the output $\{y_{K}\}$, the temperature of input configuration is determined via

[TABLE]

namely, the temperature $\alpha$ with the highest “probability” $y_{K}$ is picked as the output temperature. The training is implemented by tuning the weights and biases with the Adam optimizer kingma2014adam and the error function,

[TABLE]

which is the cross entropy between the target indicator function and output distribution function, where $i$ denotes the label of input configurations. The indicator function is defined by

[TABLE]

The convolutional neural network consists of three convolutional layers followed by the final fully-connected layer (41). We shall discuss it in more detail in Sec. III.2.

Summary of our procedure is as follows:

Step 1

Gather configurations via the standard Markov-chain Monte-Carlo method: We do not need the machine learning in this step. Data set $\{\sigma_{i}\}$ for each spin configuration at fixed $K$ is stored.

Step 2

Train the neural-network as a thermometer: The input is the spin configuration and the output is the predicted temperature.111 Because of the overlap of the energy probability distributions between each temperature, there is the upper-bound of accuracy of the prediction even if we ideally train the neural-network. See Appendix A for detail.

Step 3

Analyze trained weights and biases appearing in the neural-network: We discuss how the machine parameters contain information of the phase transition.

II.1 2D Ising model

We briefly discuss how the phase transition detection works. We first take a look at 2D Ising model, which was already studied in Ref. Tanaka:2016rtu with 100 target temperatures and its weights behave like an order parameter, i.e. spontaneous magnetization. Since our primary purpose is to understand its mechanism rather than to quantitatively estimate the critical temperature, we reduce the number of target temperatures to 20: $K=0.05,0.1,0.15,\dots,1.0$. We perform the supervised training using neural network (11), with lattice size $L=16$, the number of hidden units $N_{\mathrm{h}}=80$, and $N_{\mathrm{o}}=20$ corresponding to the 20 target temperatures. The critical temperature is exactly known to be $K^{\text{exact}}_{\text{c}}=\frac{1}{2}\ln(\sqrt{2}+1)\simeq 0.4407$ onsager1944crystal . The weights of second layer after the training is shown in Fig. 1. In Ref. Tanaka:2016rtu , the critical temperature was predicted by fitting the sum of the weights by $c_{1}\tanh[c_{2}(K-K_{\text{c}})]-c_{3}$ with free parameters $c_{1},c_{2},c_{3}$ for lattice sizes $L=8,16,32$. Indeed, the average of the final weights appears to behave like an order parameter (right panel of Fig 1). We will discuss the detailed structure of the weights in the next section.

II.2 2D 3-state Potts model

Before getting into the detail of learning mechanism of critical temperature of 2D Ising model, we take a look at another example, 2D 3-state Potts model. The Hamiltonian is given by

[TABLE]

where $\Phi_{i}$ takes three values, a generalization of Ising spin $\sigma_{i}$. Hence, configurations $\{\Phi_{i}\}$ labeled by temperatures $K$ are the inputs of neural network. The 2D 3-state Potts model is known to exhibit the second order phase transition at $K_{\text{c}}\simeq 1.0050$ because of the fluctuation unlike the simple Landau theory Wu1982 ; baxter1973potts ; alexander1975s .

After a training with the same neural network architecture as that for 2D Ising model, we obtain the weights and their average shown in Fig. 2. We again find a drastic change in the weight structure around the critical temperature.

III Discussion

So far, we have observed that the change in value of weights in the second layer signals the phase transition, which implies that the information of critical temperature is somehow encoded in the trained neural network. In what follows, we carefully examine the trained fully-connected/convolutional neural networks and attempt to understand what (physical) quantity they extract as a feature of input configurations and how it is related to temperature prediction as well as the critical temperature detection Wetzel:2017ooo ; Suchsland2018 .

III.1 Magnetization encoded in neural network

Since the order parameter of phase transition in the 2D Ising model is the spontaneous magnetization, it sounds natural that it is encoded in the neural network after the training. To give a quantitative argument we construct a simplified model by examining the weights and biases of training neural network in case of 2D Ising model. First, we reduce the number of hidden unit in (11) from 80 to 3 for the sake of simplicity. We notice that it still captures the critical temperature as shown in Fig. 3.

Before modeling the second layer, we examine the characteristics of the first layer. Figure 4 shows the correlation between the output of the first layer $\tilde{x}_{a}:=w^{(1)}_{ai}\sigma_{i}+b^{(1)}_{a}$ of Eq. (12) and magnetization density of the input Ising spin configuration. From this observation, we model these output by three lines linear in magnetization $m(\{\sigma\})$ as shown in Fig. 4b:

[TABLE]

where $\epsilon>0$ is a constant. Furthermore, as an activation function, we use the max function, that assigns 1 to the maximum entry and 0 to the rest, instead of softmax for our purpose; this replacement does not change our final result. $x_{a}=\text{max}(\tilde{x}_{a})$ yields the following vectors depending on magnetization of inputs $m$,

[TABLE]

The parameter $\epsilon$ may be interpreted as a threshold magnetization separating the ferromagnetic and paramagnetic phases carrasquilla2017machine .

Having understood the magnetization dependence of the three hidden units, we next analyze the second layer. The weights in the layer are given in Fig. 3. We divide the temperature into three pieces: low, critical, and high temperatures, respectively represented by the following vectors,

[TABLE]

in $N_{\mathrm{o}}$-dimensional output space. This procedure effectively reduces the output dimension $N_{\mathrm{o}}$ to 3. According to Fig. 3, we parametrize the weights in the following way,

[TABLE]

with $\Delta>\delta>0$. Elements of $(K\times a)$-matrix $w^{(2)}$ are the weights $w^{(2)}_{Ka}$. We neglect the biases as they are much smaller than the weights. Precise parametrization is not necessary for the following discussion. Then, $y_{K}=\text{max}(\tilde{y}_{K})=\text{max}(w^{(2)}_{Ka}x_{a}+b^{(2)}_{K})$222Remember that the max function here is defined to be a function assignning 1 to the maximum entry and 0 to the rest. yields the following output,

[TABLE]

which are predicted as low, high, and low temperature, respectively. Since the configurations with $m<-\epsilon$ and $m>\epsilon$ are in the ordered phase, they are correctly predicted as low temperature phase. The configurations with $-\epsilon\leq m<\epsilon$ is also correctly predicted as high temperature. However, it cannot detect the intermediate temperature, i.e., the critical temperature in this case. Furthermore, the different temperatures within the ordered/disordered phase are not distinguished by the trained neural network even if we introduce more than three target temperatures in Eq. (22) because upper branch of the weights (and biases) are almost temperature independent above and below $K_{\mathrm{c}}$, respectively, as seen in Fig. 3. Therefore, the trained network is capable of distinguishing only high or low temperature.

One might think that this is due to the fact that we have single threshold parameter $\epsilon$ corresponding to the three hidden unit. Actually, we can increase the number of threshold parameters by introducing more hidden units. But, it does not lead to higher resolution of temperatures. In fact, even if we increase the number of hidden units, the weights in the second layer show only two patterns of temperature dependence: most of them behave like blue and green curves and the rest behave like the orange curve in Fig. 3. This is exactly what is observed in Fig. 1. Increasing the hidden units simply results in duplicating the second layer’s weights that are already observed in Fig. 3, and consequently, the predicted temperature is either high or low no matter how many hidden units are introduced.

What we have learned from the above analyses is as follows: The network obtained the information of magnetization which manifests itself in the output of the first layer. However, it is hard for this network to discriminate each temperature except for the difference between ordered and disordered phases based on the magnetization. From the viewpoint of machine learning parameters, this is due to the fact that the weights of the second layer are temperature independent except around the critical temperature.

III.2 Energy and temperature prediction

We have found in the last subsection that the magnetization of 2D Ising model was build in the temperature-supervised fully-connected neural network, that in turn allows us to read off the critical temperature from its weight structure. While the critical temperature seems to be well detected, we have not discussed the temperature prediction itself. Interestingly, the accuracy of temperature learning can be theoretically computed to be 40.1% in case of 2D Ising model under the above setup, giving the upper bound for the accuracy of temperature prediction by machine learning Aoki2018 (see Appendix A for details). However, we note that the test accuracy of temperature prediction by fully-connected neural network is 16.8%, which is not even close to theoretically predicted accuracy 40.1%. We could train better from the viewpoint of temperature prediction although we did not need for the phase transition detection by extracting magnetization as we have already seen. What happens if we design the neural network architecture in such a way that the temperature prediction accuracy improves?

To answer the question, we use a convolutional neural network shown below, enabling us to achieve higher accuracy in two dimensional image recognition.

[TABLE]

The network has three convolutional layers (Conv2d layers from KERAS) with square filters with size $(s_{i},s_{i})$, strides $(s_{i},s_{i})$ and the number of channels $C_{i}$, each of which is followed by ReLU activation function. Then, the output is passed to fully-connected layer, whose input and output are of out interest and analyzed in detail. The output of the second last layer, denoted by $x_{a}$, is given by

[TABLE]

takes a value in $[0,\infty)$. $\tilde{x}_{a}$ is an output of the previous layer before passed to the ReLU activation. Then, the output $x_{a}$ plays a role of input of the fully-connected layer to give a prediction of temperature via (13) and (14).

Following the last subsection, we put three hidden units playing a role of input of the fully-connected layer and attempt to understand what the neural network learns as a result of the training. To this end, the parameters are set as follows:

[TABLE]

leading to the number of hidden units $N_{\mathrm{h}}=3$. With this setup we achieved 35.9% of accuracy on a test data set. It is twice higher than fully-connected layers, although still not very close to the bound partially due to the lack of statistics and very small number of hidden units.

Figures 5 shows the correlation of the output of first layer with magnetization (Fig. 5a) and internal energy (Fig. 5b), respectively. We clearly see the transition in what the neural network has learned. The first-layer outputs are now proportional to the internal energy rather than the magnetization of input configurations. Also, we notice that the weights of the second layer obtain mild dependence on temperature, implying the information of the critical temperature is blurred (Fig. 7); as we shall see momentarily, the oscillating orange line is irrelevant in the temperature prediction.

We now proceed to a simplified parametrization of the last two layers. Here, $(w^{(1)}_{ai},b^{(1)}_{a})$ and $(w^{(2)}_{Ka},b^{(2)}_{K})$ stand for weights and biases in the second last and last layer, respectively. As we have already mentioned and checked in Fig. 5b, the output of the first layer is proportional to energy $E(\{\sigma_{i}\})$ of input configuration $\{\sigma_{i}\}$. After passing to the ReLU activation,

[TABLE]

where $\epsilon$ and $\phi$ are positive constants. Domain of $E$ is restricted so that $x_{a}$ does not take a negative value.

Having observed that the input of the fully-connected layer is proportional to $E$, let us consider what would be an optimal estimation of the temperature of an input configuration $\{\sigma_{i}\}$ Aoki2018 . The probability $P(\{\sigma_{i}\};K)$ that the configuration $\{\sigma_{i}\}$ appears at temperature $K$ is given by

[TABLE]

Therefore, the likelihood that $\{\sigma_{i}\}$ is generated at temperature $K$ is represented by the following “probability”,

[TABLE]

where $F=-\ln\mathcal{Z}(K)$ is the free energy and $K^{\prime}$ is summed over the target temperatures. Then, we obtain the estimated temperature,

[TABLE]

We remark here that the free energy is a function of temperature $K$ and holds the information of phase transition although it does not exhibit genuine singularity in finite systems.

Based on the above consideration, we guess fully-connected weights of our neural network so that the resultant output behaves like $\eqref{eq:theory_output}$. Then, we compare with the actual parameters of the trained network. To this end, we parametrize the weights as follows:

[TABLE]

with constant parameters $p,q,r,s$ and a common nonlinear function $G(K)$. The orange curve in Fig. 7, $w^{(2)}_{K1}$, is not relevant to our consideration because $x_{1}=\text{ReLU}(\tilde{x}_{1})=0$. We use an observation from the simulation to neglect the bias, where it is much smaller than the value of weights.

Then, $y_{K}=\text{softmax}(w^{(2)}_{Ka}x_{a}+b^{(2)}_{K})$ with Eqs. (44) and (48) yields the following output;

[TABLE]

where we dropped $K$-independent terms because they do not affect the outcome of the max function. The expression indeed takes the form of $y_{K}^{\text{theory}}$ (46) if $-2\epsilon(\phi G(K)+pK)=(p+r)F$ is satisfied. We plot the theoretically predicted weights in Fig. 6:

[TABLE]

$F(K)$ is the free energy analytically calculated in 2D Ising model. It agrees very well with the actual weights obtained from the trained neural network shown as blue and green curves in Fig. 7. Based on these considerations, we conclude that the information of the phase transition is again encoded in the fully-connected weight because $G(K)$ in Eq. (48) is directly related to the free energy. Particularly, the critical temperature is obtained by detecting the enhancement of the second derivative of weights with respect to temperature.

It is noted that the model or the weight parametrization demonstrated above is one of many possibilities yielding the same temperature prediction. For example, the quadratic $K$-dependence could arise from the biases on the last layer instead of the weights Aoki2018 . In that case, the information of phase transition or free energy should be encoded in the bias. How the physical information is stored in the machine parameters depends on the architecture of neural network including the number of hidden units or activation functions.

Having gained the insight on the internal structure of the neural networks, we argue on why the convolutional neural network works better than the one composed only of fully-connected layers in predicting temperatures from physical viewpoint. As we have observed in this section, the latter captures the magnetization in the intermediate layer, while the former extracts the internal energy, based on which it tries to infer the temperature. However, the temperature-dependence of magnetization in the Ising model is small except around the critical temperature, compared with that of internal energy. Thus, it is plausible that the neural network extracts information of the internal energy for more accurate temperature prediction. Indeed, our convolutional neural network successfully learns the internal energy to achieve higher accuracy. The better performance by the convolutional neural network should be readily understood. It is due to the fact that the convolutional layers identify the feature of input spin configurations by exploiting their spatial structure such as spatial correlation of Ising spins. The fully-connected layers, however, are blind to the structure.

Finally, we mention that the discussion given for 2D Ising model also holds for the 3-state Potts model.

IV Conclusion

We revisited the phase transition detection of the 2D Ising model based on temperature-supervised machine learning to clarify the underlying mechanism. …

We first demonstrated that the fully-connected neural network shows the drastic change in its weight structure of the second layer as a result of training on 2D Ising model as well as 3-state Potts model. Closer look at the neural network with 3 hidden units revealed that it actually captures the magnetization in the first layer. The phase transition detection is, however, a consequence of low prediction accuracy of temperature on input configurations except around the critical temperature.

On the other hand, employing the convolutional neural network, we succeeded in improving the temperature-prediction accuracy. It turned out that the trained convolutional network captures the internal energy of input configurations instead of magnetization. Also, the weights in the last fully-connected layer do not show any drastic change in their value in contrast to the former case and this aspect is understood from the viewpoint of optimal temperature prediction. In this case, the weights are proportional to free energy, and hence, the physical information including the critical temperature is again encoded in them.

Interestingly, the trained neural network extract different physical information depending on how they are trained. A “bad” neural network in terms of temperature prediction tries to capture the magnetization of input spin configurations, which happens to be convenient for detecting critical temperature as it is the order parameter. Improvement of the network architecture allow us to construct a “good (better)” temperature predictor. But the information of phase transition is encoded in the network more implicitly.

Acknowledgment

We thank Akinori Tanaka for providing us with the code used in Tanaka:2016rtu . We are also grateful to Yutaka Yamaguti for his useful comments. K.K. is supported by the Grants-in-Aid for Scientific Research from JSPS (No. 18K03618). The work of A.T. was supported by the RIKEN Special Postdoctoral Researcher program.

Appendix A Temperature prediction and its accuracy

We think about how temperature prediction of Ising spin configuration works Aoki2018 . We start with preparing Ising spin configurations generated by Markov-chain Monte-Carlo algorithm at target temperatures. The configurations at a fixed temperature $K$ are distributed over energy with mean $\langle E\rangle$333$\langle E\rangle$ stands for a thermal expectation value of the Hamiltonian’s eigenvalues. and variance $\langle E^{2}-\langle E\rangle^{2}\rangle$, each of which is given by

[TABLE]

Given a spin configuration, we consider the optimal prediction of temperature. Energy can be calculated for each configuration. It is noted that, since the standard deviation of energy density $E/L^{2}$ is proportional to $L^{-1}$, the energy probability distribution does not admit width in infinite system. In such case, provided a certain configuration, we should be able to correctly predict the temperature at which it is generated by calculating its energy density. However, the distribution at each temperature has finite width if we consider finite systems. In that case, temperature prediction does not necessarily give the correct answer because there are overlaps between energy distributions (Fig. 8). The best we can do is to guess the temperature of the configuration by maximally likelihood estimate, namely, the temperature that is most likely to yield the configuration’s energy is the optimally predicted temperature. Consequently, there is an upper bound in the accuracy of prediction from the overlap of the energy probability distributions in finite system.

Let us take a look at two dimensional Ising model in more detail. Since it is exactly solved onsager1944crystal , we have an explicit expression of the free energy density,

[TABLE]

with the number of sites $N$. We do not incorporating the finite size effect. Then, we obtain the energy probability distributions approximated by the Gaussian distributions with mean $\langle E\rangle$ and variance $\langle(E-\langle E\rangle)^{2}\rangle$ at the temperatures of our interest. We show energy distributions at $20$ different temperatures $K=0.05,0.1,0.15,\dots,1.0$ in Fig. 9. We see that the overlaps between distributions are significant while they are reduced around the critical temperature $K_{\text{c}}^{\text{exact}}\sim 0.4407$. The accuracy obtained by maximum-likelihood estimate is as low as 40.1%, implying that a thermometer constructed by machine learning can achieve an accuracy 40.1% at most. Nevertheless, it turns out that the trained neural network can work as a phase transition detector.