Cluster-based Input Weight Initialization for Echo State Networks

TL;DR

This paper introduces a cluster-based, unsupervised method for initializing input weights in Echo State Networks using K-Means, which improves efficiency and reservoir size estimation over traditional random initialization.

Contribution

Proposes a novel K-Means based unsupervised initialization method for ESNs that enhances performance and reservoir size estimation compared to random initialization.

Findings

Comparable or better performance than random initialization

Requires fewer reservoir neurons

Enables reservoir size estimation based on data

Abstract

Echo State Networks (ESNs) are a special type of recurrent neural networks (RNNs), in which the input and recurrent connections are traditionally generated randomly, and only the output weights are trained. Despite the recent success of ESNs in various tasks of audio, image and radar recognition, we postulate that a purely random initialization is not the ideal way of initializing ESNs. The aim of this work is to propose an unsupervised initialization of the input connections using the $K$-Means algorithm on the training data. We show that for a large variety of datasets this initialization performs equivalently or superior than a randomly initialized ESN whilst needing significantly less reservoir neurons. Furthermore, we discuss that this approach provides the opportunity to estimate a suitable size of the reservoir based on prior knowledge about the data.