Self-Organization Towards $1/f$ Noise in Deep Neural Networks

TL;DR

This study demonstrates that deep neural networks trained on natural language exhibit $1/f$ noise in neuron activations, similar to biological brains, and that overcapacity leads to deviations from this pattern, indicating optimal learning signatures.

Contribution

The paper reveals the presence of $1/f$ noise in deep neural networks and links it to optimal learning, providing a new perspective on neural activity patterns in artificial systems.

Findings

$1/f$ noise observed in LSTM neuron activations

Overcapacity causes deviation from $1/f$ noise towards white noise

Resemblance between $1/f$ exponents in neural networks and human brain signals

Abstract

The presence of $1/f$ noise, also known as pink noise, is a well-established phenomenon in biological neural networks, and is thought to play an important role in information processing in the brain. In this study, we find that such $1/f$ noise is also found in deep neural networks trained on natural language, resembling that of their biological counterparts. Specifically, we trained Long Short-Term Memory (LSTM) networks on the `IMDb' AI benchmark dataset, then measured the neuron activations. The detrended fluctuation analysis (DFA) on the time series of the different neurons demonstrate clear $1/f$ patterns, which is absent in the time series of the inputs to the LSTM. Interestingly, when the neural network is at overcapacity, having more than enough neurons to achieve the learning task, the activation patterns deviate from $1/f$ noise and shifts towards white noise. This is because many of the neurons are not effectively used, showing little fluctuations when fed with input data. We further examine the exponent values in the $1/f$ noise in ``internal" and ``external" activations in the LSTM cell, finding some resemblance in the variations of the exponents in fMRI signals of the human brain. Our findings further supports the hypothesis that $1/f$ noise is a signature of optimal learning. With deep learning models approaching or surpassing humans in certain tasks, and being more ``experimentable'' than their biological counterparts, our study suggests that they are good candidates to understand the fundamental origins of $1/f$ noise.