A Safety Modulator Actor-Critic Method in Model-Free Safe Reinforcement Learning and Application in UAV Hovering

TL;DR

This paper introduces the SMAC method, combining a safety modulator and distributional critic, to enhance safety and performance in model-free safe reinforcement learning for UAV hovering tasks.

Contribution

It presents a novel safety modulator actor-critic framework with a distributional critic, specifically designed to improve safety constraint satisfaction and reduce overestimation in safe RL.

Findings

SMAC effectively maintains safety constraints during UAV hovering.

SMAC outperforms baseline algorithms in safety and reward metrics.

Experimental results validate the method in both simulation and real-world scenarios.

Abstract

This paper proposes a safety modulator actor-critic (SMAC) method to address safety constraint and overestimation mitigation in model-free safe reinforcement learning (RL). A safety modulator is developed to satisfy safety constraints by modulating actions, allowing the policy to ignore safety constraint and focus on maximizing reward. Additionally, a distributional critic with a theoretical update rule for SMAC is proposed to mitigate the overestimation of Q-values with safety constraints. Both simulation and real-world scenarios experiments on Unmanned Aerial Vehicles (UAVs) hovering confirm that the SMAC can effectively maintain safety constraints and outperform mainstream baseline algorithms.