Stochastic Minimum-Cost Reach-Avoid Reinforcement Learning

TL;DR

This paper introduces a new reinforcement learning framework that ensures probabilistic reach-avoid constraints are satisfied while minimizing costs in stochastic environments, using novel certificates and a Bellman formulation.

Contribution

The paper proposes reach-avoid probability certificates and a contraction-based Bellman approach to jointly optimize costs and satisfy probabilistic constraints in stochastic RL.

Findings

Algorithms converge to locally optimal policies.

Experiments show higher reach-avoid satisfaction rates.

Cost performance is improved in MuJoCo simulations.

Abstract

We study stochastic minimum-cost reach-avoid reinforcement learning, where an agent must satisfy a reach-avoid specification with probability at least $p$ while minimizing expected cumulative costs in stochastic environments. Existing safe and constrained reinforcement learning methods typically fail to jointly enforce probabilistic reach-avoid constraints and optimize cost in the learning setting in stochastic environments. To address this challenge, we introduce reach-avoid probability certificates (RAPCs), which identify states from which stochastic reach-avoid constraints are satisfiable. Building on RAPCs, we develop a contraction-based Bellman formulation that serves as a principled surrogate for integrating reach-avoid considerations into reinforcement learning, enabling cost optimization under probabilistic constraints. We establish almost sure convergence of the proposed algorithms to locally optimal policies with respect to the resulting objective. Experiments in the MuJoCo simulator demonstrate improved cost performance and consistently higher reach-avoid satisfaction rates.