Safe But Not Sorry: Reducing Over-Conservatism in Safety Critics via Uncertainty-Aware Modulation

TL;DR

This paper introduces the Uncertain Safety Critic (USC), a novel method that balances safety and reward in reinforcement learning by focusing conservatism on uncertain regions, significantly reducing safety violations without sacrificing performance.

Contribution

USC is a new uncertainty-aware critic training approach that improves safety-performance trade-offs in reinforcement learning by selectively applying conservatism.

Findings

Reduces safety violations by ~40%.

Decreases cost gradient prediction error by ~83%.

Maintains or improves reward performance.

Abstract

Ensuring the safe exploration of reinforcement learning (RL) agents is critical for deployment in real-world systems. Yet existing approaches struggle to strike the right balance: methods that tightly enforce safety often cripple task performance, while those that prioritize reward leave safety constraints frequently violated, producing diffuse cost landscapes that flatten gradients and stall policy improvement. We introduce the Uncertain Safety Critic (USC), a novel approach that integrates uncertainty-aware modulation and refinement into critic training. By concentrating conservatism in uncertain and costly regions while preserving sharp gradients in safe areas, USC enables policies to achieve effective reward-safety trade-offs. Extensive experiments show that USC reduces safety violations by approximately 40% while maintaining competitive or higher rewards, and reduces the error between predicted and true cost gradients by approximately 83%, breaking the prevailing trade-off between safety and performance and paving the way for scalable safe RL.