Internal Representations in Spiking Neural Networks, criticality and the Renormalization Group

TL;DR

This study investigates whether deep spiking neural networks exhibit criticality in their internal representations, using biological-inspired methods, and finds broad-tailed fluctuations without clear evidence of critical phase transitions.

Contribution

It introduces a methodology inspired by biological cortical recordings to analyze internal representations in spiking neural networks and interprets the results through a renormalization group perspective.

Findings

Broad-tailed distribution of IR fluctuations observed

No conclusive evidence of power law avalanche distributions

Fluctuations resemble non-critical ferromagnetic systems

Abstract

Optimal information processing in peripheral sensory systems has been associated in several examples to the signature of a critical or near critical state. Furthermore, cortical systems have also been described to be in a critical state in both wake and anesthetized experimental models, both {\it in vitro} and {\it in vivo}. We investigate whether a similar signature characterizes the internal representations (IR) of a multilayer (deep) spiking artificial neural network performing computationally simple but meaningful cognitive tasks, using a methodology inspired in the biological setup, with cortical implanted electrodes in rats, either freely behaving or under different levels of anesthesia. The increase of the characteristic time of the decay of the correlation of fluctuations of the IR, found when the network input changes, are indications of a broad-tailed distribution of IR fluctuations. The broad tails are present even when the network is not yet capable of performing the classification tasks, either due to partial training or to the effect of a low dose of anesthesia in a simple model. However, we don't find enough evidence of power law distributions of avalanche size and duration. We interpret the results from a renormalization group perspective to point out that despite having broad tails, this is not related to a critical transition but rather similar to fluctuations driven by the reversal of the magnetic field in a ferromagnetic system. Another example of persistent correlation of fluctuations of a non critical system is constructed, where a particle undergoes Brownian motion on a slowly varying potential.