VIP-Loco: A Visually Guided Infinite Horizon Planning Framework for Legged Locomotion

TL;DR

VIP-Loco is a novel framework that combines vision-based scene understanding with reinforcement learning and model predictive control to enable robust, adaptable, and interpretable legged robot locomotion across complex, dynamic environments.

Contribution

It introduces an integrated approach that unifies perception, learning, and planning for legged robots, bridging the gap between model-based and model-free methods.

Findings

VIP-Loco outperforms existing methods in diverse challenging environments.

The framework demonstrates effective locomotion on multiple robot types.

Ablation studies confirm the importance of integrated perception and planning.

Abstract

Perceptive locomotion for legged robots requires anticipating and adapting to complex, dynamic environments. Model Predictive Control (MPC) serves as a strong baseline, providing interpretable motion planning with constraint enforcement, but struggles with high-dimensional perceptual inputs and rapidly changing terrain. In contrast, model-free Reinforcement Learning (RL) adapts well across visually challenging scenarios but lacks planning. To bridge this gap, we propose VIP-Loco, a framework that integrates vision-based scene understanding with RL and planning. During training, an internal model maps proprioceptive states and depth images into compact kinodynamic features used by the RL policy. At deployment, the learned models are used within an infinite-horizon MPC formulation, combining adaptability with structured planning. We validate VIP-Loco in simulation on challenging locomotion tasks, including slopes, stairs, crawling, tilting, gap jumping, and climbing, across three robot morphologies: a quadruped (Unitree Go1), a biped (Cassie), and a wheeled-biped (TronA1-W). Through ablations and comparisons with state-of-the-art methods, we show that VIP-Loco unifies planning and perception, enabling robust, interpretable locomotion in diverse environments.