Interpreting Deep Learning by Establishing a Rigorous Corresponding Relationship with Renormalization Group

TL;DR

This paper establishes a rigorous theoretical connection between deep neural networks and the renormalization group in statistical mechanics, showing that neural network training parallels the RG process in extracting macroscopic features.

Contribution

It provides a formal proof linking neural network parameters to RG fixed points, offering a new interpretability framework based on statistical mechanics principles.

Findings

Neural network parameters converge to RG fixed points.

Training neural networks is mathematically equivalent to RG transformations.

The approach applies to fully connected networks and 1D Ising models.

Abstract

In this paper, we focus on the interpretability of deep neural network. Our work is motivated by the renormalization group (RG) in statistical mechanics. RG plays the role of a bridge connecting microscopical properties and macroscopic properties, the coarse graining procedure of it is quite similar with the calculation between layers in the forward propagation of the neural network algorithm. From this point of view we establish a rigorous corresponding relationship between the deep neural network (DNN) and RG. Concretely, we consider the most general fully connected network structure and real space RG of one dimensional Ising model. We prove that when the parameters of neural network achieve their optimal value, the limit of coupling constant of the output of neural network equals to the fixed point of the coupling constant in RG of one dimensional Ising model. This conclusion shows that the training process of neural network is equivalent to RG and therefore the network extract macroscopic feature from the input data just like RG.