E-SDS: Environment-aware See it, Do it, Sorted - Automated Environment-Aware Reinforcement Learning for Humanoid Locomotion

TL;DR

E-SDS is a framework that combines vision-language models with real-time terrain sensing to automatically generate reward functions, enabling robust humanoid locomotion across complex terrains with minimal manual effort.

Contribution

It introduces a novel environment-aware reward generation method that integrates VLMs with terrain sensors, improving locomotion policy robustness and reducing manual reward engineering.

Findings

Enabled successful stair descent in humanoids

Reduced velocity tracking error by up to 82.6%

Automated reward design time reduced from days to under two hours

Abstract

Vision-language models (VLMs) show promise in automating reward design in humanoid locomotion, which could eliminate the need for tedious manual engineering. However, current VLM-based methods are essentially "blind", as they lack the environmental perception required to navigate complex terrain. We present E-SDS (Environment-aware See it, Do it, Sorted), a framework that closes this perception gap. E-SDS integrates VLMs with real-time terrain sensor analysis to automatically generate reward functions that facilitate training of robust perceptive locomotion policies, grounded by example videos. Evaluated on a Unitree G1 humanoid across four distinct terrains (simple, gaps, obstacles, stairs), E-SDS uniquely enabled successful stair descent, while policies trained with manually-designed rewards or a non-perceptive automated baseline were unable to complete the task. In all terrains, E-SDS also reduced velocity tracking error by 51.9-82.6%. Our framework reduces the human effort of reward design from days to less than two hours while simultaneously producing more robust and capable locomotion policies.