Fuz-RL: A Fuzzy-Guided Robust Framework for Safe Reinforcement Learning under Uncertainty

TL;DR

Fuz-RL introduces a fuzzy measure-guided framework for safe reinforcement learning that enhances robustness and safety by estimating value functions with a novel fuzzy Bellman operator, avoiding complex min-max optimization.

Contribution

The paper presents a new fuzzy Bellman operator and proves its equivalence to distributionally robust safe RL, improving safety and performance under uncertainty without min-max optimization.

Findings

Fuz-RL effectively improves safety in various uncertain environments.

It integrates seamlessly with existing safe RL methods.

Empirical results show significant safety and control performance gains.

Abstract

Safe Reinforcement Learning (RL) is crucial for achieving high performance while ensuring safety in real-world applications. However, the complex interplay of multiple uncertainty sources in real environments poses significant challenges for interpretable risk assessment and robust decision-making. To address these challenges, we propose Fuz-RL, a fuzzy measure-guided robust framework for safe RL. Specifically, our framework develops a novel fuzzy Bellman operator for estimating robust value functions using Choquet integrals. Theoretically, we prove that solving the Fuz-RL problem (in Constrained Markov Decision Process (CMDP) form) is equivalent to solving distributionally robust safe RL problems (in robust CMDP form), effectively avoiding min-max optimization. Empirical analyses on safe-control-gym and safety-gymnasium scenarios demonstrate that Fuz-RL effectively integrates with existing safe RL baselines in a model-free manner, significantly improving both safety and control performance under various types of uncertainties in observation, action, and dynamics.