Analyzing Echo-state Networks Using Fractal Dimension

TL;DR

This paper explores the fractal nature of recurrent neural networks' hidden states, proposing that fractal dimension analysis can optimize network initialization and improve reservoir computing performance.

Contribution

It introduces the use of fractal dimension as a novel metric for analyzing and optimizing the initialization of echo-state networks, linking fractal properties to network connectivity.

Findings

Fractal dimension of hidden states is lower than the number of units.

Optimal fractal dimension is close to the number of units.

Fractal analysis can guide reservoir initialization.

Abstract

This work joins aspects of reservoir optimization, information-theoretic optimal encoding, and at its center fractal analysis. We build on the observation that, due to the recursive nature of recurrent neural networks, input sequences appear as fractal patterns in their hidden state representation. These patterns have a fractal dimension that is lower than the number of units in the reservoir. We show potential usage of this fractal dimension with regard to optimization of recurrent neural network initialization. We connect the idea of `ideal' reservoirs to lossless optimal encoding using arithmetic encoders. Our investigation suggests that the fractal dimension of the mapping from input to hidden state shall be close to the number of units in the network. This connection between fractal dimension and network connectivity is an interesting new direction for recurrent neural network initialization and reservoir computing.