Effects of sampling and horizon in predictive reinforcement learning

TL;DR

This paper investigates how sampling time and prediction horizon affect the performance of hybrid reinforcement learning and model-predictive control agents in a mobile robot parking task, revealing optimal hyper-parameter settings.

Contribution

It provides an empirical analysis of the impact of sampling and horizon hyper-parameters on hybrid RL-MPC agents in a real-world control scenario.

Findings

Sampling exhibits a 'sweet spot' behavior for optimal performance.

RL agents perform better at shorter prediction horizons.

Benchmarking against a simple MPC variant highlights the advantages of hybrid approaches.

Abstract

Plain reinforcement learning (RL) may be prone to loss of convergence, constraint violation, unexpected performance, etc. Commonly, RL agents undergo extensive learning stages to achieve acceptable functionality. This is in contrast to classical control algorithms which are typically model-based. An direction of research is the fusion of RL with such algorithms, especially model-predictive control (MPC). This, however, introduces new hyper-parameters related to the prediction horizon. Furthermore, RL is usually concerned with Markov decision processes. But the most of the real environments are not time-discrete. The factual physical setting of RL consists of a digital agent and a time-continuous dynamical system. There is thus, in fact, yet another hyper-parameter -- the agent sampling time. In this paper, we investigate the effects of prediction horizon and sampling of two hybrid RL-MPC-agents in a case study with a mobile robot parking, which is in turn a canonical control problem. We benchmark the agents with a simple variant of MPC. The sampling showed a kind of a "sweet spot" behavior, whereas the RL agents demonstrated merits at shorter horizons.