Efficient Cross-Validation of Echo State Networks

TL;DR

This paper introduces efficient cross-validation schemes for Echo State Networks, enabling accurate hyper-parameter tuning with minimal additional computational cost, often comparable to single split validation.

Contribution

It proposes several cross-validation methods for ESNs and an efficient algorithm that maintains low time complexity, improving practical hyper-parameter tuning.

Findings

Cross-validation can be performed with similar time complexity as single split validation.

The proposed methods are effective on various real-world datasets.

Space complexity remains unchanged with the new validation schemes.

Abstract

Echo State Networks (ESNs) are known for their fast and precise one-shot learning of time series. But they often need good hyper-parameter tuning for best performance. For this good validation is key, but usually, a single validation split is used. In this rather practical contribution we suggest several schemes for cross-validating ESNs and introduce an efficient algorithm for implementing them. The component that dominates the time complexity of the already quite fast ESN training remains constant (does not scale up with $k$) in our proposed method of doing $k$-fold cross-validation. The component that does scale linearly with $k$ starts dominating only in some not very common situations. Thus in many situations $k$-fold cross-validation of ESNs can be done for virtually the same time complexity as a simple single split validation. Space complexity can also remain the same. We also discuss when the proposed validation schemes for ESNs could be beneficial and empirically investigate them on several different real-world datasets.