PCHC: Enabling Preference Conditioned Humanoid Control via Multi-Objective Reinforcement Learning

TL;DR

This paper introduces PCHC, a novel multi-objective reinforcement learning framework that enables humanoid robots to adaptively balance competing goals like speed and energy efficiency through a single preference-conditioned policy.

Contribution

The paper presents a new MORL framework with a preference-conditioned policy and a Beta distribution-based alignment mechanism, allowing diverse behaviors without multiple separate policies.

Findings

Enables real-time adaptation of robot behavior based on preferences

Demonstrates effectiveness on humanoid tasks in simulation and real-world

Provides a spectrum of behaviors from a single policy

Abstract

Humanoid robots often need to balance competing objectives, such as maximizing speed while minimizing energy consumption. While current reinforcement learning (RL) methods can master complex skills like fall recovery and perceptive locomotion, they are constrained by fixed weighting strategies that produce a single suboptimal policy, rather than providing a diverse set of solutions for sophisticated multi-objective control. In this paper, we propose a novel framework leveraging Multi-Objective Reinforcement Learning (MORL) to achieve Preference-Conditioned Humanoid Control (PCHC). Unlike conventional methods that require training a series of policies to approximate the Pareto front, our framework enables a single, preference-conditioned policy to exhibit a wide spectrum of diverse behaviors. To effectively integrate these requirements, we introduce a Beta distribution-based alignment mechanism based on preference vectors modulating a Mixture-of-Experts (MoE) module. We validated our approach on two representative humanoid tasks. Extensive simulations and real-world experiments demonstrate that the proposed framework allows the robot to adaptively shift its objective priorities in real-time based on the input preference condition.