Certifying Hamilton-Jacobi Reachability Learned via Reinforcement Learning

TL;DR

This paper introduces a framework to certify Hamilton-Jacobi reachability learned via reinforcement learning by converting value errors into precise safety enclosures, ensuring correctness and safety guarantees.

Contribution

It develops a novel certification method linking RL value errors to HJ PDE offsets, enabling formal safety guarantees for learned value functions.

Findings

Successfully certifies a learned value function for a double-integrator system.

Provides formal inner and outer safety enclosures using SMT.

Demonstrates compatibility with deep reinforcement learning.

Abstract

We present a framework to \emph{certify} Hamilton--Jacobi (HJ) reachability learned by reinforcement learning (RL). Building on a discounted initial time \emph{travel-cost} formulation that makes small-step RL value iteration provably equivalent to a forward Hamilton--Jacobi (HJ) equation with damping, we convert certified learning errors into calibrated inner/outer enclosures of strict backward reachable tube. The core device is an additive-offset identity: if $W_\lambda$ solves the discounted travel-cost Hamilton--Jacobi--Bellman (HJB) equation, then $W_\varepsilon:=W_\lambda + \varepsilon$ solves the same PDE with a constant offset $\lambda\varepsilon$. This means that a uniform value error is \emph{exactly} equal to a constant HJB offset. We establish this uniform value error via two routes: (A) a Bellman operator-residual bound, and (B) a HJB PDE-slack bound. Our framework preserves HJ-level safety semantics and is compatible with deep RL. We demonstrate the approach on a double-integrator system by formally certifying, via satisfiability modulo theories (SMT), a value function learned through reinforcement learning to induce provably correct inner and outer backward-reachable set enclosures over a compact region of interest.