Online Adaptive Reinforcement Learning with Echo State Networks for Non-Stationary Dynamics

TL;DR

This paper introduces a lightweight online reinforcement learning method using Echo State Networks for rapid adaptation to changing environments, outperforming existing approaches in stability and speed without extensive pretraining.

Contribution

It presents a novel online adaptation framework with ESNs and RLS that enables fast, stable RL policy updates in non-stationary environments without backpropagation or privileged info.

Findings

Outperforms domain randomization and adaptive baselines in non-stationary tasks

Achieves stable adaptation within a few control steps

Handles intra-episode environment changes without resets

Abstract

Reinforcement learning (RL) policies trained in simulation often suffer from severe performance degradation when deployed in real-world environments due to non-stationary dynamics. While Domain Randomization (DR) and meta-RL have been proposed to address this issue, they typically rely on extensive pretraining, privileged information, or high computational cost, limiting their applicability to real-time and edge systems. In this paper, we propose a lightweight online adaptation framework for RL based on Reservoir Computing. Specifically, we integrate an Echo State Networks (ESNs) as an adaptation module that encodes recent observation histories into a latent context representation, and update its readout weights online using Recursive Least Squares (RLS). This design enables rapid adaptation without backpropagation, pretraining, or access to privileged information. We evaluate the proposed method on CartPole and HalfCheetah tasks with severe and abrupt environment changes, including periodic external disturbances and extreme friction variations. Experimental results demonstrate that the proposed approach significantly outperforms DR and representative adaptive baselines under out-of-distribution dynamics, achieving stable adaptation within a few control steps. Notably, the method successfully handles intra-episode environment changes without resetting the policy. Due to its computational efficiency and stability, the proposed framework provides a practical solution for online adaptation in non-stationary environments and is well suited for real-world robotic control and edge deployment.