Reservoir Computing for Fast, Simplified Reinforcement Learning on Memory Tasks

TL;DR

This paper demonstrates that reservoir computing simplifies and accelerates reinforcement learning on memory-dependent tasks by providing a fixed, high-dimensional memory representation that reduces training complexity and enhances generality.

Contribution

The study introduces reservoir computing as an effective alternative to trainable recurrent layers for reinforcement learning, especially in memory tasks, improving speed and simplicity.

Findings

Reservoir computing speeds up reinforcement learning on memory tasks.

It eliminates the need for backpropagation through time.

It enhances general memory capabilities for meta-learning.

Abstract

Tasks in which rewards depend upon past information not available in the current observation set can only be solved by agents that are equipped with short-term memory. Usual choices for memory modules include trainable recurrent hidden layers, often with gated memory. Reservoir computing presents an alternative, in which a recurrent layer is not trained, but rather has a set of fixed, sparse recurrent weights. The weights are scaled to produce stable dynamical behavior such that the reservoir state contains a high-dimensional, nonlinear impulse response function of the inputs. An output decoder network can then be used to map the compressive history represented by the reservoir's state to any outputs, including agent actions or predictions. In this study, we find that reservoir computing greatly simplifies and speeds up reinforcement learning on memory tasks by (1) eliminating the need for backpropagation of gradients through time, (2) presenting all recent history simultaneously to the downstream network, and (3) performing many useful and generic nonlinear computations upstream from the trained modules. In particular, these findings offer significant benefit to meta-learning that depends primarily on efficient and highly general memory systems.