Deep Q-network using reservoir computing with multi-layered readout

TL;DR

This paper enhances reservoir computing-based reinforcement learning by employing a multi-layered readout, leading to improved performance in classical control tasks without the need for backpropagation through time.

Contribution

It introduces a multi-layered readout in reservoir computing for RL, improving learning performance without BPTT, addressing computational and stability issues.

Findings

Improved performance on four classical control tasks

Multi-layered readout outperforms single-layer in learning efficiency

Reservoir computing with multi-layered readout avoids BPTT issues

Abstract

Recurrent neural network (RNN) based reinforcement learning (RL) is used for learning context-dependent tasks and has also attracted attention as a method with remarkable learning performance in recent research. However, RNN-based RL has some issues that the learning procedures tend to be more computationally expensive, and training with backpropagation through time (BPTT) is unstable because of vanishing/exploding gradients problem. An approach with replay memory introducing reservoir computing has been proposed, which trains an agent without BPTT and avoids these issues. The basic idea of this approach is that observations from the environment are input to the reservoir network, and both the observation and the reservoir output are stored in the memory. This paper shows that the performance of this method improves by using a multi-layered neural network for the readout layer, which regularly consists of a single linear layer. The experimental results show that using multi-layered readout improves the learning performance of four classical control tasks that require time-series processing.