Handling Long and Richly Constrained Tasks through Constrained Hierarchical Reinforcement Learning

TL;DR

This paper introduces CoSHRL, a hierarchical reinforcement learning method that effectively manages complex safety constraints in long-horizon decision tasks like robotic cleaning, outperforming existing constrained RL approaches.

Contribution

The paper presents CoSHRL, a hierarchical RL framework that handles complex safety constraints and flexible thresholds without retraining, advancing safety in long-term decision making.

Findings

CoSHRL outperforms existing methods in safety-constrained tasks.

It effectively manages CVaR and other complex safety constraints.

The approach adapts to different constraint thresholds without retraining.

Abstract

Safety in goal directed Reinforcement Learning (RL) settings has typically been handled through constraints over trajectories and have demonstrated good performance in primarily short horizon tasks. In this paper, we are specifically interested in the problem of solving temporally extended decision making problems such as robots cleaning different areas in a house while avoiding slippery and unsafe areas (e.g., stairs) and retaining enough charge to move to a charging dock; in the presence of complex safety constraints. Our key contribution is a (safety) Constrained Search with Hierarchical Reinforcement Learning (CoSHRL) mechanism that combines an upper level constrained search agent (which computes a reward maximizing policy from a given start to a far away goal state while satisfying cost constraints) with a low-level goal conditioned RL agent (which estimates cost and reward values to move between nearby states). A major advantage of CoSHRL is that it can handle constraints on the cost value distribution (e.g., on Conditional Value at Risk, CVaR) and can adjust to flexible constraint thresholds without retraining. We perform extensive experiments with different types of safety constraints to demonstrate the utility of our approach over leading approaches in constrained and hierarchical RL.