Reinforcement Learning in a Safety-Embedded MDP with Trajectory Optimization

TL;DR

This paper presents a novel reinforcement learning method that integrates trajectory optimization within a safety-embedded MDP framework, significantly improving safety and performance in safety-critical tasks and real-world robot applications.

Contribution

It introduces a new approach combining RL with trajectory optimization to embed safety constraints directly into the decision-making process.

Findings

Achieves higher rewards on Safety Gym tasks

Ensures near-zero safety violations during inference

Successfully deploys in a real robot box-pushing task

Abstract

Safe Reinforcement Learning (RL) plays an important role in applying RL algorithms to safety-critical real-world applications, addressing the trade-off between maximizing rewards and adhering to safety constraints. This work introduces a novel approach that combines RL with trajectory optimization to manage this trade-off effectively. Our approach embeds safety constraints within the action space of a modified Markov Decision Process (MDP). The RL agent produces a sequence of actions that are transformed into safe trajectories by a trajectory optimizer, thereby effectively ensuring safety and increasing training stability. This novel approach excels in its performance on challenging Safety Gym tasks, achieving significantly higher rewards and near-zero safety violations during inference. The method's real-world applicability is demonstrated through a safe and effective deployment in a real robot task of box-pushing around obstacles.