Conservative and Adaptive Penalty for Model-Based Safe Reinforcement Learning

TL;DR

This paper introduces CAP, a model-based safe reinforcement learning framework that uses uncertainty-based penalties and adaptive tuning to ensure safety and improve sample efficiency in real-world environments.

Contribution

CAP is a novel safe RL method that incorporates model uncertainty into penalties and adaptively adjusts them, guaranteeing safety during training.

Findings

CAP reduces safety violations compared to prior methods.

CAP improves sample efficiency in state and image-based environments.

Theoretical guarantees ensure safety of policies during training.

Abstract

Reinforcement Learning (RL) agents in the real world must satisfy safety constraints in addition to maximizing a reward objective. Model-based RL algorithms hold promise for reducing unsafe real-world actions: they may synthesize policies that obey all constraints using simulated samples from a learned model. However, imperfect models can result in real-world constraint violations even for actions that are predicted to satisfy all constraints. We propose Conservative and Adaptive Penalty (CAP), a model-based safe RL framework that accounts for potential modeling errors by capturing model uncertainty and adaptively exploiting it to balance the reward and the cost objectives. First, CAP inflates predicted costs using an uncertainty-based penalty. Theoretically, we show that policies that satisfy this conservative cost constraint are guaranteed to also be feasible in the true environment. We further show that this guarantees the safety of all intermediate solutions during RL training. Further, CAP adaptively tunes this penalty during training using true cost feedback from the environment. We evaluate this conservative and adaptive penalty-based approach for model-based safe RL extensively on state and image-based environments. Our results demonstrate substantial gains in sample-efficiency while incurring fewer violations than prior safe RL algorithms. Code is available at: https://github.com/Redrew/CAP