Illuminating the Black Box of Reservoir Computing

TL;DR

This paper investigates the internal mechanisms of reservoir computing, revealing that minimal components can suffice for certain tasks and that the readout layer often performs most of the computation, challenging traditional assumptions.

Contribution

It systematically identifies the minimal computational ingredients needed in reservoir computing, including the roles of neurons, nonlinearity, and connectivity, with insights into design factors affecting performance.

Findings

Readout layer can perform most computations in some cases.

Design parameters like input structure and activation steepness critically influence performance.

Minimal reservoir configurations can achieve task requirements with simplified structures.

Abstract

Reservoir computers, based on large recurrent neural networks with fixed random connections, are known to perform a wide range of information processing tasks. However, the nature of data transformations within the reservoir, the interplay of input matrix, reservoir, and readout layer, as well as the effect of varying design parameters remain poorly understood. In this study, we shift the focus from performance maximization to systematic simplification, aiming to identify the minimal computational ingredients required for different model tasks. We examine how many neurons, how much nonlinearity, and which connective structure is necessary and sufficient to perform certain tasks, considering also neurons with non-sigmoidal activation functions and networks with non-random connectivity. Surprisingly, we find non-trivial cases where the readout layer performs the bulk of the computation, with the reservoir merely providing weak nonlinearity and memory. Furthermore, design aspects often considered secondary, such as the structure of the input matrix, the steepness of activation functions, or the precise input/output timing, emerge as critical determinants of system performance in certain tasks.