Predictive Modeling in the Reservoir Kernel Motif Space

TL;DR

This paper introduces a reservoir kernel motif-based time series prediction method that, despite its simplicity, can outperform complex models like transformers on certain datasets, offering a new perspective on model complexity versus performance.

Contribution

The paper presents a novel reservoir kernel motif approach for time series prediction, providing a geometric interpretation and demonstrating competitive performance against state-of-the-art models.

Findings

Outperforms transformer models on univariate time series

Achieves competitive results on multivariate datasets

Simple linear reservoir models can surpass complex deep learning models

Abstract

This work proposes a time series prediction method based on the kernel view of linear reservoirs. In particular, the time series motifs of the reservoir kernel are used as representational basis on which general readouts are constructed. We provide a geometric interpretation of our approach shedding light on how our approach is related to the core reservoir models and in what way the two approaches differ. Empirical experiments then compare predictive performances of our suggested model with those of recent state-of-art transformer based models, as well as the established recurrent network model - LSTM. The experiments are performed on both univariate and multivariate time series and with a variety of prediction horizons. Rather surprisingly we show that even when linear readout is employed, our method has the capacity to outperform transformer models on univariate time series and attain competitive results on multivariate benchmark datasets. We conclude that simple models with easily controllable capacity but capturing enough memory and subsequence structure can outperform potentially over-complicated deep learning models. This does not mean that reservoir motif based models are preferable to other more complex alternatives - rather, when introducing a new complex time series model one should employ as a sanity check simple, but potentially powerful alternatives/baselines such as reservoir models or the models introduced here.