Learning a Low-dimensional Representation of a Safe Region for Safe Reinforcement Learning on Dynamical Systems

TL;DR

This paper introduces a data-driven method to efficiently learn and adapt a low-dimensional safe region representation for reinforcement learning in complex dynamical systems, demonstrated on a quadcopter example.

Contribution

It proposes a general online adaptation approach to accurately identify safe regions, enhancing the applicability of safe reinforcement learning frameworks.

Findings

More reliable safe region representations were obtained.

The approach extended safe RL to complex systems.

Demonstrated effectiveness on a quadcopter example.

Abstract

For safely applying reinforcement learning algorithms on high-dimensional nonlinear dynamical systems, a simplified system model is used to formulate a safe reinforcement learning framework. Based on the simplified system model, a low-dimensional representation of the safe region is identified and is used to provide safety estimates for learning algorithms. However, finding a satisfying simplified system model for complex dynamical systems usually requires a considerable amount of effort. To overcome this limitation, we propose in this work a general data-driven approach that is able to efficiently learn a low-dimensional representation of the safe region. Through an online adaptation method, the low-dimensional representation is updated by using the feedback data such that more accurate safety estimates are obtained. The performance of the proposed approach for identifying the low-dimensional representation of the safe region is demonstrated with a quadcopter example. The results show that, compared to previous work, a more reliable and representative low-dimensional representation of the safe region is derived, which then extends the applicability of the safe reinforcement learning framework.