PILOT: A Perceptive Integrated Low-level Controller for Loco-manipulation over Unstructured Scenes

TL;DR

PILOT introduces a unified reinforcement learning framework that integrates perceptive locomotion and manipulation for humanoid robots, improving stability and terrain handling in unstructured environments.

Contribution

It presents a novel single-stage RL approach with a cross-modal encoder and Mixture-of-Experts architecture for enhanced loco-manipulation in complex scenes.

Findings

Demonstrates superior stability and terrain traversability in simulation and real-world tests.

Achieves higher command tracking precision compared to baselines.

Validates effectiveness on the Unitree G1 humanoid robot.

Abstract

Humanoid robots hold great potential for diverse interactions and daily service tasks within human-centered environments, necessitating controllers that seamlessly integrate precise locomotion with dexterous manipulation. However, most existing whole-body controllers lack exteroceptive awareness of the surrounding environment, rendering them insufficient for stable task execution in complex, unstructured scenarios.To address this challenge, we propose PILOT, a unified single-stage reinforcement learning (RL) framework tailored for perceptive loco-manipulation, which synergizes perceptive locomotion and expansive whole-body control within a single policy. To enhance terrain awareness and ensure precise foot placement, we design a cross-modal context encoder that fuses prediction-based proprioceptive features with attention-based perceptive representations. Furthermore, we introduce a Mixture-of-Experts (MoE) policy architecture to coordinate diverse motor skills, facilitating better specialization across distinct motion patterns. Extensive experiments in both simulation and on the physical Unitree G1 humanoid robot validate the efficacy of our framework. PILOT demonstrates superior stability, command tracking precision, and terrain traversability compared to existing baselines. These results highlight its potential to serve as a robust, foundational low-level controller for loco-manipulation in unstructured scenes.