Risk-Aware Safe Reinforcement Learning for Control of Stochastic Linear Systems

TL;DR

This paper introduces a risk-aware safe reinforcement learning framework for stochastic linear systems that combines a safe controller with an RL controller, ensuring high-confidence safety and efficient control without relying on detailed system models.

Contribution

It proposes a novel method to learn safe controllers alongside RL controllers, using data-driven optimization and interpolation techniques to enhance safety and reduce data requirements.

Findings

High-confidence safety certification without high-fidelity models

Avoidance of myopic interventions and undesired equilibria

Reduced data needs and variance in safety violations

Abstract

This paper presents a risk-aware safe reinforcement learning (RL) control design for stochastic discrete-time linear systems. Rather than using a safety certifier to myopically intervene with the RL controller, a risk-informed safe controller is also learned besides the RL controller, and the RL and safe controllers are combined together. Several advantages come along with this approach: 1) High-confidence safety can be certified without relying on a high-fidelity system model and using limited data available, 2) Myopic interventions and convergence to an undesired equilibrium can be avoided by deciding on the contribution of two stabilizing controllers, and 3) highly efficient and computationally tractable solutions can be provided by optimizing over a scalar decision variable and linear programming polyhedral sets. To learn safe controllers with a large invariant set, piecewise affine controllers are learned instead of linear controllers. To this end, the closed-loop system is first represented using collected data, a decision variable, and noise. The effect of the decision variable on the variance of the safe violation of the closed-loop system is formalized. The decision variable is then designed such that the probability of safety violation for the learned closed-loop system is minimized. It is shown that this control-oriented approach reduces the data requirements and can also reduce the variance of safety violations. Finally, to integrate the safe and RL controllers, a new data-driven interpolation technique is introduced. This method aims to maintain the RL agent's optimal implementation while ensuring its safety within environments characterized by noise. The study concludes with a simulation example that serves to validate the theoretical results.