MO-Playground: Massively Parallelized Multi-Objective Reinforcement Learning for Robotics

TL;DR

This paper introduces MORLAX and MO-Playground, enabling massively parallelized multi-objective reinforcement learning for robotics, significantly reducing computation time and improving Pareto front quality in complex robotic tasks.

Contribution

The paper presents a GPU-native MORL algorithm and a GPU-accelerated environment suite, enabling rapid approximation of Pareto sets in multi-objective robotics.

Findings

Achieved 25-270x speed-ups over CPU methods.

Successfully learned Pareto-optimal locomotion policies for BRUCE robot.

Demonstrated versatility across multiple realistic objectives.

Abstract

Multi-objective reinforcement learning (MORL) is a powerful tool to learn Pareto-optimal policy families across conflicting objectives. However, unlike traditional RL algorithms, existing MORL algorithms do not effectively leverage large-scale parallelization to concurrently simulate thousands of environments, resulting in vastly increased computation time. Ultimately, this has limited MORL's application towards complex multi-objective robotics problems. To address these challenges, we present 1) MORLAX, a new GPU-native, fast MORL algorithm, and 2) MO-Playground, a pip-installable playground of GPU-accelerated multi-objective environments. Together, MORLAX and MO-Playground approximate Pareto sets within minutes, offering 25-270x speed-ups compared to legacy CPU-based approaches whilst achieving superior Pareto front hypervolumes. We demonstrate the versatility of our approach by implementing a custom BRUCE humanoid robot environment using MO-Playground and learning Pareto-optimal locomotion policies across 6 realistic objectives for BRUCE, such as smoothness, efficiency and arm swinging.