Determination of the edge of criticality in echo state networks through Fisher information maximization

TL;DR

This paper introduces an unsupervised, Fisher information-based method to identify the edge of criticality in echo state networks, optimizing their computational capabilities without relying on application-specific tuning.

Contribution

It proposes a theoretically grounded, non-parametric Fisher information estimator to determine the critical operating region of echo state networks, independent of input signals.

Findings

Fisher information peaks at the critical edge of the network.

The method accurately identifies the critical region across benchmarks.

Experimental results show improved network performance at identified critical points.

Abstract

It is a widely accepted fact that the computational capability of recurrent neural networks is maximized on the so-called "edge of criticality". Once the network operates in this configuration, it performs efficiently on a specific application both in terms of (i) low prediction error and (ii) high short-term memory capacity. Since the behavior of recurrent networks is strongly influenced by the particular input signal driving the dynamics, a universal, application-independent method for determining the edge of criticality is still missing. In this paper, we aim at addressing this issue by proposing a theoretically motivated, unsupervised method based on Fisher information for determining the edge of criticality in recurrent neural networks. It is proven that Fisher information is maximized for (finite-size) systems operating in such critical regions. However, Fisher information is notoriously difficult to compute and either requires the probability density function or the conditional dependence of the system states with respect to the model parameters. The paper takes advantage of a recently-developed non-parametric estimator of the Fisher information matrix and provides a method to determine the critical region of echo state networks, a particular class of recurrent networks. The considered control parameters, which indirectly affect the echo state network performance, are explored to identify those configurations lying on the edge of criticality and, as such, maximizing Fisher information and computational performance. Experimental results on benchmarks and real-world data demonstrate the effectiveness of the proposed method.