Reinforcement Learning for Legged Robots: Motion Imitation from Model-Based Optimal Control

TL;DR

This paper introduces MIMOC, a reinforcement learning controller for legged robots that learns agile locomotion by imitating dynamically consistent, model-based optimal control trajectories, reducing the need for fine-tuning and improving robustness.

Contribution

MIMOC is a novel RL approach that imitates reference trajectories including torque references, enhancing robustness and reducing fine-tuning compared to prior imitation methods.

Findings

MIMOC outperforms traditional model-based controllers on challenging terrains.

Imitating torque references improves policy performance.

MIMOC demonstrates successful real-world deployment on Mini-Cheetah.

Abstract

We propose MIMOC: Motion Imitation from Model-Based Optimal Control. MIMOC is a Reinforcement Learning (RL) controller that learns agile locomotion by imitating reference trajectories from model-based optimal control. MIMOC mitigates challenges faced by other motion imitation RL approaches because the references are dynamically consistent, require no motion retargeting, and include torque references. Hence, MIMOC does not require fine-tuning. MIMOC is also less sensitive to modeling and state estimation inaccuracies than model-based controllers. We validate MIMOC on the Mini-Cheetah in outdoor environments over a wide variety of challenging terrain, and on the MIT Humanoid in simulation. We show cases where MIMOC outperforms model-based optimal controllers, and show that imitating torque references improves the policy's performance.