Learning Humanoid Arm Motion via Centroidal Momentum Regularized Multi-Agent Reinforcement Learning

TL;DR

This paper introduces a multi-agent reinforcement learning framework for humanoid robots that learns coordinated arm and leg motions to improve balance and stability during various locomotion tasks, inspired by human arm swinging.

Contribution

The paper presents a novel multi-agent RL approach with centroidal momentum regularization enabling emergent arm motion for humanoid balance control, validated through simulation and real-world deployment.

Findings

Arm motions reduce overall angular momentum and ground reaction moments.

The approach outperforms single-agent and other multi-agent baselines.

Robust performance achieved across diverse locomotion tasks.

Abstract

Humans naturally swing their arms during locomotion to regulate whole-body dynamics, reduce angular momentum, and help maintain balance. Inspired by this principle, we present a limb-level multi-agent reinforcement learning (RL) framework that enables coordinated whole-body control of humanoid robots through emergent arm motion. Our approach employs separate actor-critic structures for the arms and legs, trained with centralized critics but decentralized actors that share only base states and centroidal angular momentum (CAM) observations, allowing each agent to specialize in task-relevant behaviors through modular reward design. The arm agent guided by CAM tracking and damping rewards promotes arm motions that reduce overall angular momentum and vertical ground reaction moments, contributing to improved balance during locomotion or under external perturbations. Comparative studies with single-agent and alternative multi-agent baselines further validate the effectiveness of our approach. Finally, we deploy the learned policy on a humanoid platform, achieving robust performance across diverse locomotion tasks, including flat-ground walking, rough terrain traversal, and stair climbing.