Model-Based Reinforcement Learning for Control under Time-Varying Dynamics

TL;DR

This paper addresses reinforcement learning for control in systems with changing dynamics, proposing a new algorithm that adapts to non-stationarity and improves performance on benchmarks.

Contribution

It introduces a practical optimistic model-based RL algorithm with adaptive data buffers for non-stationary environments, backed by theoretical analysis.

Findings

The proposed method outperforms existing algorithms on non-stationary control benchmarks.

Explicitly limiting outdated data improves uncertainty calibration and regret guarantees.

Gaussian process models effectively handle time-varying dynamics in RL.

Abstract

Learning-based control methods typically assume stationary system dynamics, an assumption often violated in real-world systems due to drift, wear, or changing operating conditions. We study reinforcement learning for control under time-varying dynamics. We consider a continual model-based reinforcement learning setting in which an agent repeatedly learns and controls a dynamical system whose transition dynamics evolve across episodes. We analyze the problem using Gaussian process dynamics models under frequentist variation-budget assumptions. Our analysis shows that persistent non-stationarity requires explicitly limiting the influence of outdated data to maintain calibrated uncertainty and meaningful dynamic regret guarantees. Motivated by these insights, we propose a practical optimistic model-based reinforcement learning algorithm with adaptive data buffer mechanisms and demonstrate improved performance on continuous control benchmarks with non-stationary dynamics.