World Model-based Perception for Visual Legged Locomotion

TL;DR

This paper introduces World Model-based Perception (WMP), a novel approach that builds an environment model to improve visual perception and control in legged robots, enhancing robustness and transferability from simulation to real-world terrains.

Contribution

WMP is a new method that learns a world model for perception, enabling better simulation-to-real transfer and improved locomotion performance without privileged information.

Findings

WMP outperforms state-of-the-art baselines in simulated and real-world tests.

The world model accurately predicts real-world trajectories.

WMP enhances traversability and robustness in legged locomotion.

Abstract

Legged locomotion over various terrains is challenging and requires precise perception of the robot and its surroundings from both proprioception and vision. However, learning directly from high-dimensional visual input is often data-inefficient and intricate. To address this issue, traditional methods attempt to learn a teacher policy with access to privileged information first and then learn a student policy to imitate the teacher's behavior with visual input. Despite some progress, this imitation framework prevents the student policy from achieving optimal performance due to the information gap between inputs. Furthermore, the learning process is unnatural since animals intuitively learn to traverse different terrains based on their understanding of the world without privileged knowledge. Inspired by this natural ability, we propose a simple yet effective method, World Model-based Perception (WMP), which builds a world model of the environment and learns a policy based on the world model. We illustrate that though completely trained in simulation, the world model can make accurate predictions of real-world trajectories, thus providing informative signals for the policy controller. Extensive simulated and real-world experiments demonstrate that WMP outperforms state-of-the-art baselines in traversability and robustness. Videos and Code are available at: https://wmp-loco.github.io/.