Probabilistic Counterexample Guidance for Safer Reinforcement Learning (Extended Version)

TL;DR

This paper introduces a probabilistic counterexample-guided approach to enhance safe exploration in reinforcement learning, effectively reducing safety violations during online training by leveraging abstract models and counterexamples.

Contribution

It presents a novel method that uses probabilistic counterexamples to guide safe exploration, improving safety without sacrificing reward performance.

Findings

Reduced safety violations by 40.3% compared to QL and DQN.

Achieved 29.1% fewer violations than previous methods.

Maintained comparable cumulative rewards.

Abstract

Safe exploration aims at addressing the limitations of Reinforcement Learning (RL) in safety-critical scenarios, where failures during trial-and-error learning may incur high costs. Several methods exist to incorporate external knowledge or to use proximal sensor data to limit the exploration of unsafe states. However, reducing exploration risks in unknown environments, where an agent must discover safety threats during exploration, remains challenging. In this paper, we target the problem of safe exploration by guiding the training with counterexamples of the safety requirement. Our method abstracts both continuous and discrete state-space systems into compact abstract models representing the safety-relevant knowledge acquired by the agent during exploration. We then exploit probabilistic counterexample generation to construct minimal simulation submodels eliciting safety requirement violations, where the agent can efficiently train offline to refine its policy towards minimising the risk of safety violations during the subsequent online exploration. We demonstrate our method's effectiveness in reducing safety violations during online exploration in preliminary experiments by an average of 40.3% compared with QL and DQN standard algorithms and 29.1% compared with previous related work, while achieving comparable cumulative rewards with respect to unrestricted exploration and alternative approaches.