Recursive Least Squares Policy Control with Echo State Network

TL;DR

This paper introduces two novel policy control algorithms using echo state networks with recursive least squares, incorporating mini-batch learning, regularization, and overestimation prevention to improve convergence in time-series tasks.

Contribution

The paper proposes ESNRLS-Q and ESNRLS-Sarsa algorithms that adapt RLS for ESN training in mini-batch mode with regularization and overestimation control, addressing prior limitations.

Findings

Algorithms demonstrate good convergence performance.

Effective reduction of overfitting and overestimation.

Enhanced stability in policy control tasks.

Abstract

The echo state network (ESN) is a special type of recurrent neural networks for processing the time-series dataset. However, limited by the strong correlation among sequential samples of the agent, ESN-based policy control algorithms are difficult to use the recursive least squares (RLS) algorithm to update the ESN's parameters. To solve this problem, we propose two novel policy control algorithms, ESNRLS-Q and ESNRLS-Sarsa. Firstly, to reduce the correlation of training samples, we use the leaky integrator ESN and the mini-batch learning mode. Secondly, to make RLS suitable for training ESN in mini-batch mode, we present a new mean-approximation method for updating the RLS correlation matrix. Thirdly, to prevent ESN from over-fitting, we use the L1 regularization technique. Lastly, to prevent the target state-action value from overestimation, we employ the Mellowmax method. Simulation results show that our algorithms have good convergence performance.