Safe Reinforcement Learning of Robot Trajectories in the Presence of Moving Obstacles

TL;DR

This paper introduces a reinforcement learning approach for generating collision-free robot trajectories amidst moving obstacles, utilizing a backup policy for evasive maneuvers and risk estimation methods to ensure safety in real-time applications.

Contribution

It presents a novel method combining a backup policy with risk estimation techniques to enable safe, real-time robot trajectory planning in dynamic environments.

Findings

Effective collision avoidance in deterministic environments

Reduced computational effort with data-based risk estimator

Successful real-time implementation on a physical robot

Abstract

In this paper, we present an approach for learning collision-free robot trajectories in the presence of moving obstacles. As a first step, we train a backup policy to generate evasive movements from arbitrary initial robot states using model-free reinforcement learning. When learning policies for other tasks, the backup policy can be used to estimate the potential risk of a collision and to offer an alternative action if the estimated risk is considered too high. No matter which action is selected, our action space ensures that the kinematic limits of the robot joints are not violated. We analyze and evaluate two different methods for estimating the risk of a collision. A physics simulation performed in the background is computationally expensive but provides the best results in deterministic environments. If a data-based risk estimator is used instead, the computational effort is significantly reduced, but an additional source of error is introduced. For evaluation, we successfully learn a reaching task and a basketball task while keeping the risk of collisions low. The results demonstrate the effectiveness of our approach for deterministic and stochastic environments, including a human-robot scenario and a ball environment, where no state can be considered permanently safe. By conducting experiments with a real robot, we show that our approach can generate safe trajectories in real time.