Learning-Enhanced Safeguard Control for High-Relative-Degree Systems: Robust Optimization under Disturbances and Faults

TL;DR

This paper introduces a novel safe reinforcement learning framework that combines high-order control barrier functions with gradient-based techniques to ensure safety and improve performance in nonlinear systems with high-relative-degree constraints and disturbances.

Contribution

It proposes the high-order reciprocal control barrier function (HO-RCBF) and gradient similarity concepts to effectively handle high-relative-degree constraints during safe learning.

Findings

Successfully addresses high-relative-degree constraints in simulations.

Enhances safety robustness under disturbances and faults.

Improves system performance while maintaining safety guarantees.

Abstract

Merely pursuing performance may adversely affect the safety, while a conservative policy for safe exploration will degrade the performance. How to balance the safety and performance in learning-based control problems is an interesting yet challenging issue. This paper aims to enhance system performance with safety guarantee in solving the reinforcement learning (RL)-based optimal control problems of nonlinear systems subject to high-relative-degree state constraints and unknown time-varying disturbance/actuator faults. First, to combine control barrier functions (CBFs) with RL, a new type of CBFs, termed high-order reciprocal control barrier function (HO-RCBF) is proposed to deal with high-relative-degree constraints during the learning process. Then, the concept of gradient similarity is proposed to quantify the relationship between the gradient of safety and the gradient of performance. Finally, gradient manipulation and adaptive mechanisms are introduced in the safe RL framework to enhance the performance with a safety guarantee. Two simulation examples illustrate that the proposed safe RL framework can address high-relative-degree constraint, enhance safety robustness and improve system performance.