Scale-invariant Feature Extraction of Neural Network and Renormalization Group Flow

TL;DR

This paper investigates the relationship between deep neural network feature extraction and renormalization group flow by analyzing an RBM trained on Ising model configurations, revealing a flow towards the critical temperature.

Contribution

It demonstrates how an unsupervised RBM trained on spin configurations exhibits a flow toward the critical point, offering insights into neural network feature extraction and RG concepts.

Findings

RBM flow approaches critical temperature T_c=2.27

RBM trained on various temperatures learns to extract features

Flow behavior is opposite to typical RG flow in Ising model

Abstract

Theoretical understanding of how deep neural network (DNN) extracts features from input images is still unclear, but it is widely believed that the extraction is performed hierarchically through a process of coarse-graining. It reminds us of the basic concept of renormalization group (RG) in statistical physics. In order to explore possible relations between DNN and RG, we use the Restricted Boltzmann machine (RBM) applied to Ising model and construct a flow of model parameters (in particular, temperature) generated by the RBM. We show that the unsupervised RBM trained by spin configurations at various temperatures from $T=0$ to $T=6$ generates a flow along which the temperature approaches the critical value $T_c=2.27$. This behavior is opposite to the typical RG flow of the Ising model. By analyzing various properties of the weight matrices of the trained RBM, we discuss why it flows towards $T_c$ and how the RBM learns to extract features of spin configurations.