Benchmarking Actor-Critic Deep Reinforcement Learning Algorithms for Robotics Control with Action Constraints

TL;DR

This paper introduces a comprehensive benchmark for action-constrained reinforcement learning algorithms in robotics, evaluating their performance across various environments and revealing insights into their effectiveness and safety considerations.

Contribution

It provides the first extensive benchmark for action-constrained RL in robotics, including evaluation of existing and novel algorithms with open-source code.

Findings

Straightforward baseline approaches can be surprisingly effective.

Evaluation across multiple environments reveals diverse algorithm strengths.

The benchmark facilitates future research in safe and feasible robotic control.

Abstract

This study presents a benchmark for evaluating action-constrained reinforcement learning (RL) algorithms. In action-constrained RL, each action taken by the learning system must comply with certain constraints. These constraints are crucial for ensuring the feasibility and safety of actions in real-world systems. We evaluate existing algorithms and their novel variants across multiple robotics control environments, encompassing multiple action constraint types. Our evaluation provides the first in-depth perspective of the field, revealing surprising insights, including the effectiveness of a straightforward baseline approach. The benchmark problems and associated code utilized in our experiments are made available online at github.com/omron-sinicx/action-constrained-RL-benchmark for further research and development.