Convolutional Reservoir Computing for World Models

TL;DR

This paper introduces a reinforcement learning model using convolutional reservoir computing that rapidly extracts features without training, achieving state-of-the-art results with minimal computational cost.

Contribution

The study presents a novel RCRC model combining fixed-weight CNNs and reservoir computing, enabling fast feature extraction and high performance without extensive training data.

Findings

Achieves state-of-the-art scores in RL tasks

Fast feature extraction with fixed-weight networks

Simple networks like single dense layer reach high scores

Abstract

Recently, reinforcement learning models have achieved great success, completing complex tasks such as mastering Go and other games with higher scores than human players. Many of these models collect considerable data on the tasks and improve accuracy by extracting visual and time-series features using convolutional neural networks (CNNs) and recurrent neural networks, respectively. However, these networks have very high computational costs because they need to be trained by repeatedly using a large volume of past playing data. In this study, we propose a novel practical approach called reinforcement learning with convolutional reservoir computing (RCRC) model. The RCRC model has several desirable features: 1. it can extract visual and time-series features very fast because it uses random fixed-weight CNN and the reservoir computing model; 2. it does not require the training data to be stored because it extracts features without training and decides action with evolution strategy. Furthermore, the model achieves state of the art score in the popular reinforcement learning task. Incredibly, we find the random weight-fixed simple networks like only one dense layer network can also reach high score in the RL task.