Continual Learning with Echo State Networks

TL;DR

This paper explores continual learning in Echo State Networks, demonstrating their potential to learn sequential data without forgetting, and highlighting benefits of CL strategies unique to ESNs.

Contribution

It introduces the application of continual learning to ESNs, providing the first evaluation of catastrophic forgetting in these fixed-recurrent models.

Findings

ESNs exhibit resilience to catastrophic forgetting

CL strategies improve ESN performance in streaming data

ESNs are promising for real-time continual learning scenarios

Abstract

Continual Learning (CL) refers to a learning setup where data is non stationary and the model has to learn without forgetting existing knowledge. The study of CL for sequential patterns revolves around trained recurrent networks. In this work, instead, we introduce CL in the context of Echo State Networks (ESNs), where the recurrent component is kept fixed. We provide the first evaluation of catastrophic forgetting in ESNs and we highlight the benefits in using CL strategies which are not applicable to trained recurrent models. Our results confirm the ESN as a promising model for CL and open to its use in streaming scenarios.