Reservoir Memory Machines as Neural Computers

TL;DR

This paper introduces an efficient echo state network with explicit memory that can recognize all regular languages and performs comparably to fully-trained models on benchmark tasks, offering a more trainable alternative to differentiable neural computers.

Contribution

It presents a novel extension of echo state networks with explicit memory, enabling recognition of all regular languages and reducing training complexity.

Findings

Recognizes all regular languages, including those unrecognizable by contractive echo state networks.

Performs comparably to deep neural computer models on benchmark tasks.

Offers a more trainable and efficient alternative to differentiable neural computers.

Abstract

Differentiable neural computers extend artificial neural networks with an explicit memory without interference, thus enabling the model to perform classic computation tasks such as graph traversal. However, such models are difficult to train, requiring long training times and large datasets. In this work, we achieve some of the computational capabilities of differentiable neural computers with a model that can be trained very efficiently, namely an echo state network with an explicit memory without interference. This extension enables echo state networks to recognize all regular languages, including those that contractive echo state networks provably can not recognize. Further, we demonstrate experimentally that our model performs comparably to its fully-trained deep version on several typical benchmark tasks for differentiable neural computers.