Safe reinforcement learning for multi-energy management systems with known constraint functions

TL;DR

This paper introduces two novel safe reinforcement learning methods, SafeFallback and GiveSafe, which ensure constraint satisfaction during training and deployment in multi-energy systems without complex optimization, improving safety and efficiency.

Contribution

The paper presents two new safe RL algorithms that decouple safety constraints from the RL formulation, providing hard guarantees without solving additional optimization problems.

Findings

SafeFallback outperforms vanilla RL in utility (102.9% vs 100%)

Both methods achieve higher utility than benchmarks (94.6% and 82.8%)

Methods provide safety guarantees even with random policies.

Abstract

Reinforcement learning (RL) is a promising optimal control technique for multi-energy management systems. It does not require a model a priori - reducing the upfront and ongoing project-specific engineering effort and is capable of learning better representations of the underlying system dynamics. However, vanilla RL does not provide constraint satisfaction guarantees - resulting in various potentially unsafe interactions within its safety-critical environment. In this paper, we present two novel safe RL methods, namely SafeFallback and GiveSafe, where the safety constraint formulation is decoupled from the RL formulation. These provide hard-constraint, rather than soft- and chance-constraint, satisfaction guarantees both during training a (near) optimal policy (which involves exploratory and exploitative, i.e. greedy, steps) as well as during deployment of any policy (e.g. random agents or offline trained RL agents). This without the need of solving a mathematical program, resulting in less computational power requirements and a more flexible constraint function formulation (no derivative information is required). In a simulated multi-energy systems case study we have shown that both methods start with a significantly higher utility (i.e. useful policy) compared to a vanilla RL benchmark and Optlayer benchmark (94,6% and 82,8% compared to 35,5% and 77,8%) and that the proposed SafeFallback method even can outperform the vanilla RL benchmark (102,9% to 100%). We conclude that both methods are viably safety constraint handling techniques applicable beyond RL, as demonstrated with random policies while still providing hard-constraint guarantees.