Use of recurrent infomax to improve the memory capability of input-driven recurrent neural networks

TL;DR

This paper demonstrates that optimizing recurrent neural networks with recurrent infomax enhances their short-term memory by increasing mutual information and inducing a delay-line structure, improving their ability to process temporal input streams.

Contribution

The study introduces the use of recurrent infomax to optimize RNNs, resulting in improved memory capabilities and the emergence of a delay-line structure.

Findings

Optimized RNNs exhibit superior short-term memory.

Recurrent infomax induces a delay-line structure.

Mutual information maximization enhances information processing.

Abstract

The inherent transient dynamics of recurrent neural networks (RNNs) have been exploited as a computational resource in input-driven RNNs. However, the information processing capability varies from RNN to RNN, depending on their properties. Many authors have investigated the dynamics of RNNs and their relevance to the information processing capability. In this study, we present a detailed analysis of the information processing capability of an RNN optimized by recurrent infomax (RI), which is an unsupervised learning scheme that maximizes the mutual information of RNNs by adjusting the connection strengths of the network. Thus, we observe that a delay-line structure emerges from the RI and the network optimized by the RI possesses superior short-term memory, which is the ability to store the temporal information of the input stream in its transient dynamics.