Dynamic Policy Learning for Legged Robot with Simplified Model Pretraining and Model Homotopy Transfer

TL;DR

This paper presents a continuation-based learning framework that pretrains policies on simplified models and gradually transfers them to complex full-body models for dynamic legged robot motions, improving efficiency and stability.

Contribution

It introduces a novel model homotopy transfer method combined with simplified model pretraining for efficient dynamic policy learning in legged robots.

Findings

Faster convergence in dynamic motion tasks

Enhanced stability during policy transfer

Successful deployment on real quadrupedal robot

Abstract

Generating dynamic motions for legged robots remains a challenging problem. While reinforcement learning has achieved notable success in various legged locomotion tasks, producing highly dynamic behaviors often requires extensive reward tuning or high-quality demonstrations. Leveraging reduced-order models can help mitigate these challenges. However, the model discrepancy poses a significant challenge when transferring policies to full-body dynamics environments. In this work, we introduce a continuation-based learning framework that combines simplified model pretraining and model homotopy transfer to efficiently generate and refine complex dynamic behaviors. First, we pretrain the policy using a single rigid body model to capture core motion patterns in a simplified environment. Next, we employ a continuation strategy to progressively transfer the policy to the full-body environment, minimizing performance loss. To define the continuation path, we introduce a model homotopy from the single rigid body model to the full-body model by gradually redistributing mass and inertia between the trunk and legs. The proposed method not only achieves faster convergence but also demonstrates superior stability during the transfer process compared to baseline methods. Our framework is validated on a range of dynamic tasks, including flips and wall-assisted maneuvers, and is successfully deployed on a real quadrupedal robot.