CReF: Cross-modal and Recurrent Fusion for Depth-conditioned Humanoid Locomotion

TL;DR

CReF is a novel depth-conditioned humanoid locomotion framework that learns directly from raw depth data, enabling robust traversal over complex terrains without relying on explicit geometric representations.

Contribution

It introduces a single-stage fusion method combining cross-modal attention and recurrent integration to improve terrain interaction and zero-shot transfer in humanoid locomotion.

Findings

Demonstrates robust traversal over diverse terrains in simulation and real-world scenarios.

Achieves effective zero-shot transfer to complex outdoor environments.

Outperforms prior methods that rely on explicit geometric abstractions.

Abstract

Stable traversal over geometrically complex terrain increasingly requires exteroceptive perception, yet prior perceptive humanoid locomotion methods often remain tied to explicit geometric abstractions, either by mediating control through robot-centric 2.5D terrain representations or by shaping depth learning with auxiliary geometry-related targets. Such designs inherit the representational bias of the intermediate or supervisory target and can be restrictive for vertical structures, perforated obstacles, and complex real-world clutter. We propose CReF (Cross-modal and Recurrent Fusion), a single-stage depth-conditioned humanoid locomotion framework that learns locomotion-relevant features directly from raw forward-facing depth without explicit geometric intermediates. CReF couples proprioception and depth tokens through proprioception-queried cross-modal attention, fuses the resulting representation with a gated residual fusion block, and performs temporal integration with a Gated Recurrent Unit (GRU) regulated by a highway-style output gate for state-dependent blending of recurrent and feedforward features. To further improve terrain interaction, we introduce a terrain-aware foothold placement reward that extracts supportable foothold candidates from foot-end point-cloud samples and rewards touchdown locations that lie close to the nearest supportable candidate. Experiments in simulation and on a physical humanoid demonstrate robust traversal over diverse terrains and effective zero-shot transfer to real-world scenes containing handrails, hollow pallet assemblies, severe reflective interference, and visually cluttered outdoor surroundings.