Safety-Aware Reinforcement Learning for Control via Risk-Sensitive Action-Value Iteration and Quantile Regression

TL;DR

This paper introduces a safety-aware reinforcement learning algorithm that uses risk-sensitive quantile regression and CVaR to improve safety and performance in stochastic control tasks.

Contribution

It proposes a novel risk-regularized quantile-based RL method with theoretical guarantees and practical benefits for safety-critical applications.

Findings

Achieves higher goal success rates in simulations.

Reduces collision rates compared to risk-neutral methods.

Provides convergence guarantees for the risk-sensitive Bellman operator.

Abstract

Mainstream approximate action-value iteration reinforcement learning (RL) algorithms suffer from overestimation bias, leading to suboptimal policies in high-variance stochastic environments. Quantile-based action-value iteration methods reduce this bias by learning a distribution of the expected cost-to-go using quantile regression. However, ensuring that the learned policy satisfies safety constraints remains a challenge when these constraints are not explicitly integrated into the RL framework. Existing methods often require complex neural architectures or manual tradeoffs due to combined cost functions. To address this, we propose a risk-regularized quantile-based algorithm integrating Conditional Value-at-Risk (CVaR) to enforce safety without complex architectures. We also provide theoretical guarantees on the contraction properties of the risk-sensitive distributional Bellman operator in Wasserstein space, ensuring convergence to a unique cost distribution. Simulations of a mobile robot in a dynamic reach-avoid task show that our approach leads to more goal successes, fewer collisions, and better safety-performance trade-offs than risk-neutral methods.