Value Functions are Control Barrier Functions: Verification of Safe Policies using Control Theory

TL;DR

This paper introduces a novel method that leverages control theory to verify the safety of reinforcement learning policies by linking value functions to control barrier functions, enabling scalable safety guarantees.

Contribution

It formalizes the connection between value functions and control barrier functions, providing new verification metrics and practical implementation strategies for safe RL policies.

Findings

Established theoretical links between value functions and control barrier functions.

Developed new metrics for verifying value functions in safety-critical tasks.

Demonstrated practical implementation for improved safe policy learning.

Abstract

Guaranteeing safe behaviour of reinforcement learning (RL) policies poses significant challenges for safety-critical applications, despite RL's generality and scalability. To address this, we propose a new approach to apply verification methods from control theory to learned value functions. By analyzing task structures for safety preservation, we formalize original theorems that establish links between value functions and control barrier functions. Further, we propose novel metrics for verifying value functions in safe control tasks and practical implementation details to improve learning. Our work presents a novel method for certificate learning, which unlocks a diversity of verification techniques from control theory for RL policies, and marks a significant step towards a formal framework for the general, scalable, and verifiable design of RL-based control systems. Code and videos are available at this https url: https://rl-cbf.github.io/