Safe Model-Based Reinforcement Learning for Systems with Parametric Uncertainties

TL;DR

This paper introduces a safe model-based reinforcement learning method for deterministic nonlinear systems with parametric uncertainties, enabling learning of constrained optimal policies without strict excitation requirements.

Contribution

It develops a novel filtered concurrent learning approach combined with barrier transformation to learn unknown parameters and control policies simultaneously.

Findings

Successfully learns approximate constrained optimal policies.

Handles parametric uncertainties without strict excitation.

Ensures safety in safety-critical systems.

Abstract

Reinforcement learning has been established over the past decade as an effective tool to find optimal control policies for dynamical systems, with recent focus on approaches that guarantee safety during the learning and/or execution phases. In general, safety guarantees are critical in reinforcement learning when the system is safety-critical and/or task restarts are not practically feasible. In optimal control theory, safety requirements are often expressed in terms of state and/or control constraints. In recent years, reinforcement learning approaches that rely on persistent excitation have been combined with a barrier transformation to learn the optimal control policies under state constraints. To soften the excitation requirements, model-based reinforcement learning methods that rely on exact model knowledge have also been integrated with the barrier transformation framework. The objective of this paper is to develop safe reinforcement learning method for deterministic nonlinear systems, with parametric uncertainties in the model, to learn approximate constrained optimal policies without relying on stringent excitation conditions. To that end, a model-based reinforcement learning technique that utilizes a novel filtered concurrent learning method, along with a barrier transformation, is developed in this paper to realize simultaneous learning of unknown model parameters and approximate optimal state-constrained control policies for safety-critical systems.