Evaluation of Constrained Reinforcement Learning Algorithms for Legged Locomotion

TL;DR

This paper explores the use of Constrained Markov Decision Processes in deep reinforcement learning for legged robots, emphasizing the importance of physical constraints for successful sim-to-real transfer and operational accuracy.

Contribution

It introduces a CMDP-based approach for constrained policy optimization in legged robot control, demonstrating improved real-world performance over traditional unconstrained RL methods.

Findings

Physical constraints are crucial for effective sim-to-real transfer.

CMDP simplifies training by separating constraints from rewards.

Constrained RL reduces operational errors in physical robot experiments.

Abstract

Shifting from traditional control strategies to Deep Reinforcement Learning (RL) for legged robots poses inherent challenges, especially when addressing real-world physical constraints during training. While high-fidelity simulations provide significant benefits, they often bypass these essential physical limitations. In this paper, we experiment with the Constrained Markov Decision Process (CMDP) framework instead of the conventional unconstrained RL for robotic applications. We perform a comparative study of different constrained policy optimization algorithms to identify suitable methods for practical implementation. Our robot experiments demonstrate the critical role of incorporating physical constraints, yielding successful sim-to-real transfers, and reducing operational errors on physical systems. The CMDP formulation streamlines the training process by separately handling constraints from rewards. Our findings underscore the potential of constrained RL for the effective development and deployment of learned controllers in robotics.