Hybrid Backpropagation Parallel Reservoir Networks

TL;DR

This paper introduces a hybrid reservoir neural network that combines fixed random reservoirs with deep neural readouts, outperforming traditional RNNs like LSTMs and GRUs on complex time series tasks.

Contribution

The paper presents a novel hybrid network architecture, HBP-ESN, integrating reservoir computing with deep learning, and introduces a meta-ring structure to reduce memory usage.

Findings

HBP-ESN outperforms LSTMs and GRUs on gesture and emotion recognition datasets.

The HBP-ESN M-Ring achieves similar performance with significantly less memory.

The hybrid approach offers a promising alternative for temporal data learning.

Abstract

In many real-world applications, fully-differentiable RNNs such as LSTMs and GRUs have been widely deployed to solve time series learning tasks. These networks train via Backpropagation Through Time, which can work well in practice but involves a biologically unrealistic unrolling of the network in time for gradient updates, are computationally expensive, and can be hard to tune. A second paradigm, Reservoir Computing, keeps the recurrent weight matrix fixed and random. Here, we propose a novel hybrid network, which we call Hybrid Backpropagation Parallel Echo State Network (HBP-ESN) which combines the effectiveness of learning random temporal features of reservoirs with the readout power of a deep neural network with batch normalization. We demonstrate that our new network outperforms LSTMs and GRUs, including multi-layer "deep" versions of these networks, on two complex real-world multi-dimensional time series datasets: gesture recognition using skeleton keypoints from ChaLearn, and the DEAP dataset for emotion recognition from EEG measurements. We show also that the inclusion of a novel meta-ring structure, which we call HBP-ESN M-Ring, achieves similar performance to one large reservoir while decreasing the memory required by an order of magnitude. We thus offer this new hybrid reservoir deep learning paradigm as a new alternative direction for RNN learning of temporal or sequential data.