Sparsity in Reservoir Computing Neural Networks

TL;DR

This paper empirically investigates how sparsity in reservoir computing neural networks influences the richness of temporal representations, highlighting the importance of input-reservoir sparsity for memory and dimensionality.

Contribution

It provides new insights into the role of sparsity in RC networks, especially in input connections, for enhancing temporal memory and representation complexity.

Findings

Sparsity in input-reservoir connections enhances temporal memory.

Sparsity increases the dimension of internal representations.

Input-reservoir sparsity is crucial for richer temporal dynamics.

Abstract

Reservoir Computing (RC) is a well-known strategy for designing Recurrent Neural Networks featured by striking efficiency of training. The crucial aspect of RC is to properly instantiate the hidden recurrent layer that serves as dynamical memory to the system. In this respect, the common recipe is to create a pool of randomly and sparsely connected recurrent neurons. While the aspect of sparsity in the design of RC systems has been debated in the literature, it is nowadays understood mainly as a way to enhance the efficiency of computation, exploiting sparse matrix operations. In this paper, we empirically investigate the role of sparsity in RC network design under the perspective of the richness of the developed temporal representations. We analyze both sparsity in the recurrent connections, and in the connections from the input to the reservoir. Our results point out that sparsity, in particular in input-reservoir connections, has a major role in developing internal temporal representations that have a longer short-term memory of past inputs and a higher dimension.