Kinematics-Aware Multi-Policy Reinforcement Learning for Force-Capable Humanoid Loco-Manipulation

TL;DR

This paper introduces a kinematics-aware multi-policy reinforcement learning framework for humanoid robots, enabling effective force interaction and loco-manipulation in industrial scenarios through a decoupled training pipeline and specialized reward strategies.

Contribution

It presents a novel three-stage reinforcement learning approach with kinematics priors and force curriculum learning for improved humanoid loco-manipulation.

Findings

Faster policy convergence due to kinematics priors

Enhanced force regulation capabilities

Effective loco-manipulation in industrial tasks

Abstract

Humanoid robots, with their human-like morphology, hold great potential for industrial applications. However, existing loco-manipulation methods primarily focus on dexterous manipulation, falling short of the combined requirements for dexterity and proactive force interaction in high-load industrial scenarios. To bridge this gap, we propose a reinforcement learning-based framework with a decoupled three-stage training pipeline, consisting of an upper-body policy, a lower-body policy, and a delta-command policy. To accelerate upper-body training, a heuristic reward function is designed. By implicitly embedding forward kinematics priors, it enables the policy to converge faster and achieve superior performance. For the lower body, a force-based curriculum learning strategy is developed, enabling the robot to actively exert and regulate interaction forces with the environment.