Linear Reservoir: A Diagonalization-Based Optimization

TL;DR

This paper presents a diagonalization-based optimization for Linear Echo State Networks that significantly reduces computational complexity while maintaining accuracy, enabling faster training and inference.

Contribution

It introduces three methods leveraging eigenbasis diagonalization to optimize Linear ESNs, improving efficiency without sacrificing predictive performance.

Findings

Achieves O(N) complexity in reservoir updates

Maintains accuracy comparable to standard Linear ESNs

Provides methods for efficient training and sampling of eigenvalues

Abstract

We introduce a diagonalization-based optimization for Linear Echo State Networks (ESNs) that reduces the per-step computational complexity of reservoir state updates from O(N^2) to O(N). By reformulating reservoir dynamics in the eigenbasis of the recurrent matrix, the recurrent update becomes a set of independent element-wise operations, eliminating the matrix multiplication. We further propose three methods to use our optimization depending on the situation: (i) Eigenbasis Weight Transformation (EWT), which preserves the dynamics of standard and trained Linear ESNs, (ii) End-to-End Eigenbasis Training (EET), which directly optimizes readout weights in the transformed space and (iii) Direct Parameter Generation (DPG), that bypasses matrix diagonalization by directly sampling eigenvalues and eigenvectors, achieving comparable performance than standard Linear ESNs. Across all experiments, both our methods preserve predictive accuracy while offering significant computational speedups, making them a replacement of standard Linear ESNs computations and training, and suggesting a shift of paradigm in linear ESN towards the direct selection of eigenvalues.