Echo State Networks for Time Series Forecasting: Hyperparameter Sweep and Benchmarking

TL;DR

This study evaluates Echo State Networks for univariate time series forecasting, demonstrating their competitive accuracy and efficiency compared to traditional models through extensive hyperparameter tuning and benchmarking.

Contribution

It provides a comprehensive hyperparameter analysis and benchmarking of ESNs against standard forecasting methods on the M4 dataset, highlighting their practical viability.

Findings

ESNs perform comparably to ARIMA and TBATS for monthly data.

For quarterly data, ESNs achieve the lowest mean MASE.

High leakage rates are generally preferred across series frequencies.

Abstract

This paper investigates the performance of Echo State Networks (ESNs) for univariate time series forecasting using a subset of the M4 Forecasting Competition dataset. Focusing on monthly and quarterly time series, we evaluate whether a simple autoregressive ESN can serve as a competitive alternative to widely used forecasting methods. The study adopts a two-stage approach: a *Parameter* dataset is used to conduct an extensive hyperparameter sweep covering leakage rate, spectral radius, reservoir size, and information criteria for regularization, resulting in over four million ESN model fits; a disjoint *Forecast* dataset is then used for out-of-sample accuracy assessment. Forecast accuracy is measured using mean absolute scaled error (MASE) and symmetric mean absolute percentage error (sMAPE) and benchmarked against simple benchmarks like drift and seasonal naive and statistical models like autoregressive integrated moving average (ARIMA), exponential smoothing state space (ETS), the Theta method, and TBATS (trigonometric, Box-Cox transformation, ARMA errors, trend, and seasonal components). The hyperparameter analysis reveals broadly consistent and interpretable patterns, with monthly series favoring moderately persistent reservoirs and quarterly series favoring more contractive dynamics. Across both frequencies, high leakage rates are preferred, while optimal spectral radii and reservoir sizes vary with frequency. In the out-of-sample benchmarking, the ESN performs on par with ARIMA and TBATS for monthly data and achieves the lowest mean MASE for quarterly data, while requiring lower computational cost than ARIMA and TBATS. Overall, the results demonstrate that ESNs offer a balance between forecast accuracy, robustness, and computational efficiency, positioning them as a practical option for time series forecasting.