Learning Spring Mass Locomotion: Guiding Policies with a Reduced-Order Model

TL;DR

This paper presents a hierarchical control approach combining reduced-order models with reinforcement learning to enable dynamic, versatile walking behaviors in legged robots, demonstrated on a human-scale biped.

Contribution

It introduces a novel control framework that integrates high-level planning with low-level learned policies for robust, adaptable legged locomotion.

Findings

Policies can generate a range of walking motions from reduced-order models.

The learned controller balances motion tracking with stability.

Demonstrated on a human-scale biped at speeds up to 1.2 m/s.

Abstract

In this paper, we describe an approach to achieve dynamic legged locomotion on physical robots which combines existing methods for control with reinforcement learning. Specifically, our goal is a control hierarchy in which highest-level behaviors are planned through reduced-order models, which describe the fundamental physics of legged locomotion, and lower level controllers utilize a learned policy that can bridge the gap between the idealized, simple model and the complex, full order robot. The high-level planner can use a model of the environment and be task specific, while the low-level learned controller can execute a wide range of motions so that it applies to many different tasks. In this letter we describe this learned dynamic walking controller and show that a range of walking motions from reduced-order models can be used as the command and primary training signal for learned policies. The resulting policies do not attempt to naively track the motion (as a traditional trajectory tracking controller would) but instead balance immediate motion tracking with long term stability. The resulting controller is demonstrated on a human scale, unconstrained, untethered bipedal robot at speeds up to 1.2 m/s. This letter builds the foundation of a generic, dynamic learned walking controller that can be applied to many different tasks.