Safe Policy Optimization via Control Barrier Function-based Safety Filters

TL;DR

This paper introduces a policy optimization method for linear systems that enhances the stability of safety filters based on control barrier functions, ensuring safety and stability during training.

Contribution

It develops a joint parameterization and optimization framework that guarantees stability constraints are satisfied throughout training, improving safety filter performance.

Findings

Removes undesired equilibria in safety filters

Improves convergence behavior of safety-filtered controllers

Maintains forward invariance of the safe set during optimization

Abstract

Control barrier function (CBF)-based safety filters provide a systematic way to enforce state constraints, but they can significantly alter the closed-loop dynamics induced by a nominal, stabilizing controller. In particular, the resulting safety-filtered system may exhibit undesirable behaviors including limit cycles, unbounded trajectories, and undesired equilibria. This paper develops a policy optimization framework to maximally enhance the stability properties of safety-filtered controllers. Focusing on linear systems with linear nominal controllers, we jointly parameterize the nominal feedback gain and safety-filter components, and optimize them using trajectory-based objectives computed from closed-loop rollouts. To ensure that the nominal controller remains stabilizing throughout training, we encode Lyapunov-based stability conditions as smooth scalar constraints and enforce them using robust safe gradient flow. This guarantees feasibility of the stability constraints along the optimization iterates and therefore avoids instability during training. Numerical experiments on obstacle-avoidance problems show that the proposed approach can remove asymptotically stable undesired equilibria and improve convergence behavior while maintaining forward invariance of the safe set.