Deep Whole-body Parkour

TL;DR

This paper introduces a unified framework that combines perception and motion control, enabling humanoid robots to perform dynamic, multi-contact tasks on uneven terrain by integrating exteroceptive sensing into whole-body motion tracking.

Contribution

It presents a novel approach that unites perceptive locomotion and general motion tracking, allowing robots to perform complex, dynamic tasks on unstructured terrain.

Findings

Enables robust vaulting and dive-rolling on uneven terrain

Trains a single policy for multiple motions across varied terrain

Significantly expands robot traversability beyond simple walking

Abstract

Current approaches to humanoid control generally fall into two paradigms: perceptive locomotion, which handles terrain well but is limited to pedal gaits, and general motion tracking, which reproduces complex skills but ignores environmental capabilities. This work unites these paradigms to achieve perceptive general motion control. We present a framework where exteroceptive sensing is integrated into whole-body motion tracking, permitting a humanoid to perform highly dynamic, non-locomotion tasks on uneven terrain. By training a single policy to perform multiple distinct motions across varied terrestrial features, we demonstrate the non-trivial benefit of integrating perception into the control loop. Our results show that this framework enables robust, highly dynamic multi-contact motions, such as vaulting and dive-rolling, on unstructured terrain, significantly expanding the robot's traversability beyond simple walking or running. https://project-instinct.github.io/deep-whole-body-parkour