Guiding Safe Exploration with Weakest Preconditions

TL;DR

This paper introduces SPICE, a neurosymbolic method for safe exploration in reinforcement learning that uses symbolic weakest preconditions to ensure safety constraints are met during training, achieving fewer violations and theoretical convergence guarantees.

Contribution

The paper presents SPICE, a novel neurosymbolic approach employing symbolic weakest preconditions for safer exploration in reinforcement learning, with theoretical convergence guarantees.

Findings

Achieves fewer safety violations compared to existing methods.

Maintains comparable performance to current safe learning techniques.

Converges to the optimal safe policy under certain assumptions.

Abstract

In reinforcement learning for safety-critical settings, it is often desirable for the agent to obey safety constraints at all points in time, including during training. We present a novel neurosymbolic approach called SPICE to solve this safe exploration problem. SPICE uses an online shielding layer based on symbolic weakest preconditions to achieve a more precise safety analysis than existing tools without unduly impacting the training process. We evaluate the approach on a suite of continuous control benchmarks and show that it can achieve comparable performance to existing safe learning techniques while incurring fewer safety violations. Additionally, we present theoretical results showing that SPICE converges to the optimal safe policy under reasonable assumptions.