Integrating Model-Based Footstep Planning with Model-Free Reinforcement Learning for Dynamic Legged Locomotion

TL;DR

This paper presents a hybrid control framework combining model-based footstep planning with reinforcement learning to enable stable and adaptable dynamic legged locomotion, validated on a humanoid robot in various terrains and maneuvers.

Contribution

It introduces a novel integration of physics-informed footstep planning with RL, enhancing adaptability and stability in legged robot locomotion.

Findings

Achieved up to 1.5 m/s walking speed on treadmill.

Successfully performed 90° and 180° turns.

Demonstrated robustness on uneven terrain.

Abstract

In this work, we introduce a control framework that combines model-based footstep planning with Reinforcement Learning (RL), leveraging desired footstep patterns derived from the Linear Inverted Pendulum (LIP) dynamics. Utilizing the LIP model, our method forward predicts robot states and determines the desired foot placement given the velocity commands. We then train an RL policy to track the foot placements without following the full reference motions derived from the LIP model. This partial guidance from the physics model allows the RL policy to integrate the predictive capabilities of the physics-informed dynamics and the adaptability characteristics of the RL controller without overfitting the policy to the template model. Our approach is validated on the MIT Humanoid, demonstrating that our policy can achieve stable yet dynamic locomotion for walking and turning. We further validate the adaptability and generalizability of our policy by extending the locomotion task to unseen, uneven terrain. During the hardware deployment, we have achieved forward walking speeds of up to 1.5 m/s on a treadmill and have successfully performed dynamic locomotion maneuvers such as 90-degree and 180-degree turns.