CAPSULE: Control-Theoretic Action Perturbations for Safe Uncertainty-Aware Reinforcement Learning

TL;DR

This paper introduces a safe reinforcement learning framework that uses probabilistic control-affine models and control barrier functions to ensure safety during exploration in complex systems.

Contribution

It develops a novel offline learning approach to construct safety constraints that incorporate model uncertainty for real-time safe action correction.

Findings

Achieves comparable task performance to existing methods.

Significantly reduces safety violations during exploration.

Demonstrates effectiveness on nonlinear continuous-control benchmarks.

Abstract

Ensuring safe exploration in high-dimensional systems with unknown dynamics remains a significant challenge. Existing safe reinforcement learning methods often provide safety guarantees only in expectation, which can still lead to safety violations. Control-theoretic approaches, in contrast, offer hard constraint-based safety guarantees but typically assume access to known system dynamics or require accurate estimation of control-affine models. In this paper, we propose a safe reinforcement learning framework that learns a probabilistic control-affine dynamics model in an offline setting. The learned model is leveraged to explicitly construct control barrier functions (CBFs) that incorporate model uncertainty to provide conservative safety constraints. These CBF constraints are enforced through an online constraint-based action correction mechanism, enabling safe exploration without overly restricting task performance. Empirical evaluations on nonlinear, complex continuous-control benchmarks demonstrate that our approach achieves returns comparable to those of existing baselines while significantly reducing safety violations.