Stage-Wise Reward Shaping for Acrobatic Robots: A Constrained Multi-Objective Reinforcement Learning Approach

TL;DR

This paper presents a stage-wise reward shaping method using constrained multi-objective reinforcement learning to simplify complex reward design for acrobatic robots, demonstrating improved performance in simulation and real-world tasks.

Contribution

It introduces a novel stage-wise reward shaping framework with a practical CMORL algorithm for complex robotic tasks, enhancing reward design and task segmentation.

Findings

Successfully applied to various acrobatic tasks in simulation and real-world environments.

Outperforms existing RL and constrained RL algorithms in task execution.

Provides a practical implementation with publicly available code.

Abstract

As the complexity of tasks addressed through reinforcement learning (RL) increases, the definition of reward functions also has become highly complicated. We introduce an RL method aimed at simplifying the reward-shaping process through intuitive strategies. Initially, instead of a single reward function composed of various terms, we define multiple reward and cost functions within a constrained multi-objective RL (CMORL) framework. For tasks involving sequential complex movements, we segment the task into distinct stages and define multiple rewards and costs for each stage. Finally, we introduce a practical CMORL algorithm that maximizes objectives based on these rewards while satisfying constraints defined by the costs. The proposed method has been successfully demonstrated across a variety of acrobatic tasks in both simulation and real-world environments. Additionally, it has been shown to successfully perform tasks compared to existing RL and constrained RL algorithms. Our code is available at https://github.com/rllab-snu/Stage-Wise-CMORL.