LeVERB: Humanoid Whole-Body Control with Latent Vision-Language Instruction

TL;DR

LeVERB introduces a hierarchical framework for humanoid whole-body control using vision-language instructions, enabling zero-shot task success and bridging the gap between semantic understanding and dynamic control in robotics.

Contribution

It presents the first sim-to-real benchmark for humanoid vision-language control and a novel hierarchical latent instruction-following framework called LeVERB.

Findings

Achieves 80% success in simple visual navigation tasks.

Attains 58.5% success rate overall, outperforming naive methods.

Demonstrates effective zero-shot generalization in humanoid control.

Abstract

Vision-language-action (VLA) models have demonstrated strong semantic understanding and zero-shot generalization, yet most existing systems assume an accurate low-level controller with hand-crafted action "vocabulary" such as end-effector pose or root velocity. This assumption confines prior work to quasi-static tasks and precludes the agile, whole-body behaviors required by humanoid whole-body control (WBC) tasks. To capture this gap in the literature, we start by introducing the first sim-to-real-ready, vision-language, closed-loop benchmark for humanoid WBC, comprising over 150 tasks from 10 categories. We then propose LeVERB: Latent Vision-Language-Encoded Robot Behavior, a hierarchical latent instruction-following framework for humanoid vision-language WBC, the first of its kind. At the top level, a vision-language policy learns a latent action vocabulary from synthetically rendered kinematic demonstrations; at the low level, a reinforcement-learned WBC policy consumes these latent verbs to generate dynamics-level commands. In our benchmark, LeVERB can zero-shot attain a 80% success rate on simple visual navigation tasks, and 58.5% success rate overall, outperforming naive hierarchical whole-body VLA implementation by 7.8 times.