Model-Based Data-Efficient and Robust Reinforcement Learning

TL;DR

This paper introduces a model-based reinforcement learning method that enhances data efficiency and robustness by combining system dynamics modeling with a two-level control optimization, outperforming traditional approaches in energy savings.

Contribution

It presents a novel two-level control framework that integrates system dynamics learning with optimization, significantly improving data efficiency and robustness over existing methods.

Findings

Reduces energy consumption more effectively than existing RL methods.

Achieves over 100-fold reduction in evaluated time steps.

Demonstrates robustness against load disturbances and model errors.

Abstract

A data-efficient learning-based control design method is proposed in this paper. It is based on learning a system dynamics model that is then leveraged in a two-level procedure. On the higher level, a simple but powerful optimization procedure is performed such that, for example, energy consumption in a vehicle can be reduced when hard state and action constraints are also introduced. Load disturbances and model errors are compensated for by a feedback controller on the lower level. In that regard, we briefly examine the robustness of both model-free and model-based learning approaches, and it is shown that the model-free approach greatly suffers from the inclusion of unmodeled dynamics. In evaluating the proposed method, it is assumed that a path is given, while the velocity and acceleration can be modified such that energy is saved, while still keeping speed limits and completion time. Compared with two well-known actor-critic reinforcement learning strategies, the suggested learning-based approach saves more energy and reduces the number of evaluated time steps by a factor of 100 or more.